Rollback Strategies

When deployments cause issues, having a solid rollback strategy minimizes business impact and restores normal operations quickly.

---

Rollback Decision Flow

┌─────────────────────────────────────────────────────────────────────────────┐
│                    ROLLBACK DECISION FLOW                                    │
└─────────────────────────────────────────────────────────────────────────────┘

     ┌──────────────────┐
     │  Issue Detected  │
     └────────┬─────────┘
              │
              ▼
     ┌──────────────────────────────────────┐
     │  Is business process blocked?         │
     └────────┬─────────────────────┬────────┘
              │ YES                 │ NO
              ▼                     ▼
     ┌──────────────────┐   ┌──────────────────┐
     │  IMMEDIATE       │   │  Can fix quickly? │
     │  ROLLBACK        │   └────────┬─────────┘
     └──────────────────┘            │
                                YES ─┤─ NO
                                     │    │
                              ┌──────┴┐  ┌┴──────────┐
                              │ HOTFIX│  │ SCHEDULED │
                              │       │  │ ROLLBACK  │
                              └───────┘  └───────────┘


     SEVERITY LEVELS:
     ─────────────────
     CRITICAL: Business stopped, data corruption → Immediate rollback
     HIGH: Major feature broken, workaround exists → Rollback within 1 hour
     MEDIUM: Minor feature affected → Hotfix or scheduled rollback
     LOW: Cosmetic issue → Fix in next release

---

Rollback Methods

Method 1: Disable Script Deployment

Fastest method - stops script from executing without code changes.

┌─────────────────────────────────────────────────────────────────────────────┐
│                    DISABLE DEPLOYMENT                                        │
└─────────────────────────────────────────────────────────────────────────────┘

     1. Navigate to: Customization → Scripting → Script Deployments
     2. Find problematic script
     3. Click Edit
     4. Change Status: Testing (or set Inactive)
     5. Save

     Result: Script stops executing for all users immediately

Via SuiteCloud CLI:

# Update deployment XML
# Change: <status>RELEASED</status>
# To: <status>TESTING</status>

# Redeploy just the deployment
suitecloud object:deploy --scriptid customdeploy_problem_script

Method 2: Redeploy Previous Version

Recommended - restores known-good code.

# If you have backup folder
cd backup_20240115
suitecloud project:deploy --account "Production"

# Or use version control
git checkout v1.2.0
suitecloud project:deploy --account "Production"

Method 3: Import from Production

If no backup exists - import objects that weren't changed.

# Create new project for rollback
suitecloud project:create --projectname rollback_temp

# Import current (broken) objects to see structure
suitecloud project:import --objects "customscript_problem"

# Manually restore previous logic in files
# Then deploy
suitecloud project:deploy

Method 4: Delete and Recreate

Last resort - for completely broken objects.

Customization → Scripting → Scripts → [Find Script] → Delete

Note: This removes execution history. Only use if necessary.

---

Quick Disable Patterns

User Event Script

/**
 * Emergency disable flag
 */
const beforeSubmit = (context) => {
    // EMERGENCY: Disable script temporarily
    const DISABLED = true;
    if (DISABLED) {
        log.audit('Script Disabled', 'Emergency disable active');
        return;
    }

    // Normal logic below...
};

Scheduled Script

const execute = (context) => {
    // Check disable parameter
    const script = runtime.getCurrentScript();
    const disabled = script.getParameter({ name: 'custscript_emergency_disable' });

    if (disabled) {
        log.audit('Script Disabled', 'Skipping execution - emergency disable');
        return;
    }

    // Normal logic...
};

Client Script

/**
 * Add disable check that can be controlled via deployment parameter
 */
const pageInit = (context) => {
    // Check if script should run
    if (window.DISABLE_CLIENT_VALIDATION === true) {
        console.log('Client validation disabled');
        return;
    }

    // Normal logic...
};

---

Rollback Checklist

┌─────────────────────────────────────────────────────────────────────────────┐
│                    ROLLBACK CHECKLIST                                        │
└─────────────────────────────────────────────────────────────────────────────┘

     BEFORE ROLLBACK
     ☐ Document the issue clearly
     ☐ Notify stakeholders
     ☐ Identify affected objects
     ☐ Locate backup/previous version
     ☐ Plan rollback steps

     DURING ROLLBACK
     ☐ Disable problematic scripts first (fastest impact)
     ☐ Deploy previous versions
     ☐ Verify deployment successful
     ☐ Test critical functionality
     ☐ Monitor execution logs

     AFTER ROLLBACK
     ☐ Confirm business processes restored
     ☐ Notify stakeholders of resolution
     ☐ Document what happened
     ☐ Create incident report
     ☐ Plan fix for next deployment

---

Partial Rollback

When only some objects need rollback:

Identify Affected Objects

// Check execution logs for errors
// Customization → Scripting → Script Execution Logs
// Filter: Today, Error level

// Typical error patterns:
// - "Cannot read property of undefined"
// - "You have exceeded the max governance"
// - "Invalid reference"

Rollback Specific Script

# Rollback single script
git show HEAD~1:src/FileCabinet/SuiteScripts/my_script.js > src/FileCabinet/SuiteScripts/my_script.js

# Deploy only that script
suitecloud object:deploy --scriptid customscript_my_script

Keep Working Parts

<!-- deploy.xml - Deploy only what needs rollback -->
<deploy>
    <objects>
        <path>~/Objects/customscript_problem_script.xml</path>
        <!-- Don't include working scripts -->
    </objects>
</deploy>

---

Data Recovery Scenarios

Scenario: Bad Data Created by Script

┌─────────────────────────────────────────────────────────────────────────────┐
│                    DATA RECOVERY WORKFLOW                                    │
└─────────────────────────────────────────────────────────────────────────────┘

     1. STOP THE BLEEDING
        └── Disable script immediately

     2. IDENTIFY AFFECTED RECORDS
        └── Search for records created/modified since deployment

     3. ASSESS DAMAGE
        ├── How many records affected?
        ├── What data is wrong?
        └── Is original data recoverable?

     4. RECOVERY OPTIONS
        ├── Manual correction (small number)
        ├── Correction script (large number)
        └── Restore from backup (if available)

Find Affected Records

/**
 * Find records modified since deployment
 */
const findAffectedRecords = (deploymentTime) => {
    return search.create({
        type: search.Type.TRANSACTION,
        filters: [
            ['lastmodifieddate', 'onorafter', deploymentTime],
            'AND',
            ['custbody_affected_field', 'isnotempty', '']
        ],
        columns: ['internalid', 'tranid', 'lastmodifieddate', 'lastmodifiedby']
    });
};

Correction Script

/**
 * @NApiVersion 2.1
 * @NScriptType MapReduceScript
 * @description Fix affected records
 */
define(['N/search', 'N/record', 'N/log'], (search, record, log) => {

    const getInputData = () => {
        // Find records to fix
        return search.create({
            type: search.Type.SALES_ORDER,
            filters: [
                ['lastmodifieddate', 'within', '1/15/2024', '1/15/2024'],
                'AND',
                ['custbody_bad_value', 'isnotempty', '']
            ]
        });
    };

    const map = (context) => {
        const data = JSON.parse(context.value);

        try {
            record.submitFields({
                type: record.Type.SALES_ORDER,
                id: data.id,
                values: {
                    'custbody_bad_value': ''  // Clear bad value
                }
            });

            log.audit('Fixed', `Order ${data.id}`);
        } catch (e) {
            log.error('Fix Failed', `Order ${data.id}: ${e.message}`);
        }
    };

    const summarize = (summary) => {
        log.audit('Complete', `Fixed ${summary.mapSummary.keys.length} records`);
    };

    return { getInputData, map, summarize };
});

---

Prevention Strategies

Version Control

# Tag releases before production deploy
git tag -a v1.2.0 -m "Production release 1.2.0"
git push origin v1.2.0

# Easy rollback to any version
git checkout v1.1.0
suitecloud project:deploy

Backup Before Deploy

#!/bin/bash
# pre_deploy_backup.sh

DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="backups/$DATE"

mkdir -p $BACKUP_DIR

# Import current production state
suitecloud project:import --objects "customscript_*" --destinationfolder $BACKUP_DIR

echo "Backup saved to $BACKUP_DIR"

Feature Flags

/**
 * Feature flag for safe rollout
 */
const FEATURE_FLAGS = {
    newValidation: false,  // Set true after verification
    enhancedEmail: false,
    v2Api: false
};

const beforeSubmit = (context) => {
    if (FEATURE_FLAGS.newValidation) {
        return newValidationLogic(context);
    }
    return oldValidationLogic(context);
};

Gradual Rollout

/**
 * Roll out to percentage of users
 */
const shouldUseNewFeature = (userId) => {
    const script = runtime.getCurrentScript();
    const rolloutPercent = script.getParameter({ name: 'custscript_rollout_percent' }) || 0;

    // Simple hash-based distribution
    const userHash = userId % 100;
    return userHash < rolloutPercent;
};

---

Incident Documentation

Incident Report Template

# Incident Report: [Brief Description]

## Summary
- **Date/Time**: 2024-01-15 18:30 PST
- **Duration**: 45 minutes
- **Severity**: High
- **Affected**: Invoice processing

## Timeline
- 18:00 - Deployment completed
- 18:15 - First error reported
- 18:20 - Issue confirmed
- 18:25 - Rollback initiated
- 18:30 - Previous version deployed
- 18:35 - Functionality restored
- 18:45 - Stakeholders notified

## Root Cause
The updated validation logic failed when processing invoices
with multiple currencies due to missing null check.

## Impact
- 15 invoices failed to save
- 3 users affected
- No data loss

## Resolution
Rolled back to version 1.2.0

## Prevention
- Add null check for currency field
- Add test case for multi-currency invoices
- Improve code review process

## Action Items
1. [ ] Fix validation logic
2. [ ] Add unit tests
3. [ ] Re-test in sandbox
4. [ ] Schedule new deployment

---

Emergency Contacts & Procedures

┌─────────────────────────────────────────────────────────────────────────────┐
│                    EMERGENCY PROCEDURES                                      │
└─────────────────────────────────────────────────────────────────────────────┘

     STEP 1: ASSESS
     └── Is this critical? (business stopped, data at risk)

     STEP 2: COMMUNICATE
     ├── Notify team lead immediately
     ├── Alert affected users
     └── Don't troubleshoot alone if critical

     STEP 3: STABILIZE
     ├── Disable problematic script
     ├── Deploy previous version
     └── Verify business can continue

     STEP 4: INVESTIGATE
     ├── Review execution logs
     ├── Identify root cause
     └── Document findings

     STEP 5: RESOLVE
     ├── Fix the issue
     ├── Test thoroughly
     └── Plan next deployment

---

Rollback Testing

Test Your Rollback Process

Before production deployment, verify you can rollback:

# 1. Deploy to sandbox
suitecloud project:deploy --account "Sandbox"

# 2. Make intentional breaking change
# 3. Deploy breaking change

# 4. Practice rollback
git checkout HEAD~1
suitecloud project:deploy --account "Sandbox"

# 5. Verify rollback worked
# 6. Document any issues with process

---

Summary

Situation	Best Rollback Method
Script causing errors	Disable deployment (fastest)
Wrong business logic	Deploy previous version
Data being corrupted	Disable + correction script
New feature not working	Feature flag off
Complete failure	Full project rollback

---

Next Steps

• Deploy to Production - Production best practices
• Testing Strategies - Prevent rollback needs
• Error Handling - Graceful error management