Skip to main content

Rollback Strategies

When deployments cause issues, having a solid rollback strategy minimizes business impact and restores normal operations quickly.

Rollback Decision Flow​

QuestionAnswerAction
Is business process blocked?YESIMMEDIATE ROLLBACK
Is business process blocked?NO, but can't fix quicklySCHEDULED ROLLBACK
Can fix quickly?YESHOTFIX

SEVERITY LEVELS:

  • CRITICAL: Business stopped, data corruption → Immediate rollback
  • HIGH: Major feature broken, workaround exists → Rollback within 1 hour
  • MEDIUM: Minor feature affected → Hotfix or scheduled rollback
  • LOW: Cosmetic issue → Fix in next release

Rollback Methods​

Method 1: Disable Script Deployment​

Fastest method - stops script from executing without code changes.

Steps:

  1. Navigate to: Customization → Scripting → Script Deployments
  2. Find problematic script
  3. Click Edit
  4. Change Status to Testing or set Inactive
  5. Save → Script stops executing immediately

Via SuiteCloud CLI:

# Update deployment XML
# Change: <status>RELEASED</status>
# To: <status>TESTING</status>

# Redeploy just the deployment
suitecloud object:deploy --scriptid customdeploy_problem_script

Method 2: Redeploy Previous Version​

Recommended - restores known-good code.

# If you have backup folder
cd backup_20240115
suitecloud project:deploy --account "Production"

# Or use version control
git checkout v1.2.0
suitecloud project:deploy --account "Production"

Method 3: Import from Production​

If no backup exists - import objects that weren't changed.

# Create new project for rollback
suitecloud project:create --projectname rollback_temp

# Import current (broken) objects to see structure
suitecloud project:import --objects "customscript_problem"

# Manually restore previous logic in files
# Then deploy
suitecloud project:deploy

Method 4: Delete and Recreate​

Last resort - for completely broken objects.

Customization → Scripting → Scripts → [Find Script] → Delete

Note: This removes execution history. Only use if necessary.

Quick Disable Patterns​

User Event Script​

/**
 * Emergency disable flag
 */
const beforeSubmit = (context) => {
    // EMERGENCY: Disable script temporarily
    const DISABLED = true;
    if (DISABLED) {
        log.audit('Script Disabled', 'Emergency disable active');
        return;
    }

    // Normal logic below...
};

Scheduled Script​

const execute = (context) => {
    // Check disable parameter
    const script = runtime.getCurrentScript();
    const disabled = script.getParameter({ name: 'custscript_emergency_disable' });

    if (disabled) {
        log.audit('Script Disabled', 'Skipping execution - emergency disable');
        return;
    }

    // Normal logic...
};

Client Script​

/**
 * Add disable check that can be controlled via deployment parameter
 */
const pageInit = (context) => {
    // Check if script should run
    if (window.DISABLE_CLIENT_VALIDATION === true) {
        console.log('Client validation disabled');
        return;
    }

    // Normal logic...
};

Rollback Checklist​

Before Rollback​

  • ☐ Document the issue clearly
  • ☐ Notify stakeholders
  • ☐ Identify affected objects
  • ☐ Locate backup/previous version
  • ☐ Plan rollback steps

During Rollback​

  • ☐ Disable problematic scripts first (fastest impact)
  • ☐ Deploy previous versions
  • ☐ Verify deployment successful
  • ☐ Test critical functionality
  • ☐ Monitor execution logs

After Rollback​

  • ☐ Confirm business processes restored
  • ☐ Notify stakeholders of resolution
  • ☐ Document what happened
  • ☐ Create incident report
  • ☐ Plan fix for next deployment

Partial Rollback​

When only some objects need rollback:

Identify Affected Objects​

// Check execution logs for errors
// Customization → Scripting → Script Execution Logs
// Filter: Today, Error level

// Typical error patterns:
// - "Cannot read property of undefined"
// - "You have exceeded the max governance"
// - "Invalid reference"

Rollback Specific Script​

# Rollback single script
git show HEAD~1:src/FileCabinet/SuiteScripts/my_script.js > src/FileCabinet/SuiteScripts/my_script.js

# Deploy only that script
suitecloud object:deploy --scriptid customscript_my_script

Keep Working Parts​

<!-- deploy.xml - Deploy only what needs rollback -->
<deploy>
    <objects>
        <path>~/Objects/customscript_problem_script.xml</path>
        <!-- Don't include working scripts -->
    </objects>
</deploy>

Data Recovery Scenarios​

Scenario: Bad Data Created by Script​

StepAction
1. STOP THE BLEEDINGDisable script immediately
2. IDENTIFY AFFECTED RECORDSSearch for records created/modified since deployment
3. ASSESS DAMAGEHow many records? What data is wrong? Is original data recoverable?
4. RECOVERY OPTIONSManual correction (small #), Correction script (large #), Restore from backup

Find Affected Records​

/**
 * Find records modified since deployment
 */
const findAffectedRecords = (deploymentTime) => {
    return search.create({
        type: search.Type.TRANSACTION,
        filters: [
            ['lastmodifieddate', 'onorafter', deploymentTime],
            'AND',
            ['custbody_affected_field', 'isnotempty', '']
        ],
        columns: ['internalid', 'tranid', 'lastmodifieddate', 'lastmodifiedby']
    });
};

Correction Script​

/**
 * @NApiVersion 2.1
 * @NScriptType MapReduceScript
 * @description Fix affected records
 */
define(['N/search', 'N/record', 'N/log'], (search, record, log) => {

    const getInputData = () => {
        // Find records to fix
        return search.create({
            type: search.Type.SALES_ORDER,
            filters: [
                ['lastmodifieddate', 'within', '1/15/2024', '1/15/2024'],
                'AND',
                ['custbody_bad_value', 'isnotempty', '']
            ]
        });
    };

    const map = (context) => {
        const data = JSON.parse(context.value);

        try {
            record.submitFields({
                type: record.Type.SALES_ORDER,
                id: data.id,
                values: {
                    'custbody_bad_value': ''  // Clear bad value
                }
            });

            log.audit('Fixed', `Order ${data.id}`);
        } catch (e) {
            log.error('Fix Failed', `Order ${data.id}: ${e.message}`);
        }
    };

    const summarize = (summary) => {
        log.audit('Complete', `Fixed ${summary.mapSummary.keys.length} records`);
    };

    return { getInputData, map, summarize };
});

Prevention Strategies​

Version Control​

# Tag releases before production deploy
git tag -a v1.2.0 -m "Production release 1.2.0"
git push origin v1.2.0

# Easy rollback to any version
git checkout v1.1.0
suitecloud project:deploy

Backup Before Deploy​

#!/bin/bash
# pre_deploy_backup.sh

DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="backups/$DATE"

mkdir -p $BACKUP_DIR

# Import current production state
suitecloud project:import --objects "customscript_*" --destinationfolder $BACKUP_DIR

echo "Backup saved to $BACKUP_DIR"

Feature Flags​

/**
 * Feature flag for safe rollout
 */
const FEATURE_FLAGS = {
    newValidation: false,  // Set true after verification
    enhancedEmail: false,
    v2Api: false
};

const beforeSubmit = (context) => {
    if (FEATURE_FLAGS.newValidation) {
        return newValidationLogic(context);
    }
    return oldValidationLogic(context);
};

Gradual Rollout​

/**
 * Roll out to percentage of users
 */
const shouldUseNewFeature = (userId) => {
    const script = runtime.getCurrentScript();
    const rolloutPercent = script.getParameter({ name: 'custscript_rollout_percent' }) || 0;

    // Simple hash-based distribution
    const userHash = userId % 100;
    return userHash < rolloutPercent;
};

Incident Documentation​

Incident Report Template​

# Incident Report: [Brief Description]

## Summary
- **Date/Time**: 2024-01-15 18:30 PST
- **Duration**: 45 minutes
- **Severity**: High
- **Affected**: Invoice processing

## Timeline
- 18:00 - Deployment completed
- 18:15 - First error reported
- 18:20 - Issue confirmed
- 18:25 - Rollback initiated
- 18:30 - Previous version deployed
- 18:35 - Functionality restored
- 18:45 - Stakeholders notified

## Root Cause
The updated validation logic failed when processing invoices
with multiple currencies due to missing null check.

## Impact
- 15 invoices failed to save
- 3 users affected
- No data loss

## Resolution
Rolled back to version 1.2.0

## Prevention
- Add null check for currency field
- Add test case for multi-currency invoices
- Improve code review process

## Action Items
1. [ ] Fix validation logic
2. [ ] Add unit tests
3. [ ] Re-test in sandbox
4. [ ] Schedule new deployment

Emergency Contacts & Procedures​

StepActionDetails
1. ASSESSIs this critical?Business stopped, data at risk
2. COMMUNICATENotify stakeholdersTeam lead, affected users, don't troubleshoot alone if critical
3. STABILIZEStop the bleedingDisable script, deploy previous version, verify business can continue
4. INVESTIGATEFind root causeReview execution logs, identify root cause, document findings
5. RESOLVEFix properlyFix the issue, test thoroughly, plan next deployment

Rollback Testing​

Test Your Rollback Process​

Before production deployment, verify you can rollback:

# 1. Deploy to sandbox
suitecloud project:deploy --account "Sandbox"

# 2. Make intentional breaking change
# 3. Deploy breaking change

# 4. Practice rollback
git checkout HEAD~1
suitecloud project:deploy --account "Sandbox"

# 5. Verify rollback worked
# 6. Document any issues with process

Summary​

SituationBest Rollback Method
Script causing errorsDisable deployment (fastest)
Wrong business logicDeploy previous version
Data being corruptedDisable + correction script
New feature not workingFeature flag off
Complete failureFull project rollback

Next Steps​