How a single execution substrate powers automations, snap-kits, keyrings, snap-in commands, AirSync extractors, and AI agent skills — without duplicating infrastructure.
---
The Problem With Building Six Execution Engines
Most platforms that need to run custom logic start with one use case and grow from there. Webhooks first. Then scheduled jobs. Then AI agent skills. Then data sync connectors. Then OAuth verification. Then dynamic UI rendering.
Before long, you have six execution engines with six deployment pipelines, six security models, and six on-call rotations.
DevRev took a different path. Every piece of custom logic — whether it's a workflow automation, a snap-kit action, a keyring verifier, a snap-in command, or an AirSync extractor — runs through one service: the Function service. One execution substrate, one security model, one deployment pipeline.
---
The Six Modalities
The Function service is the execution backbone behind all custom logic in DevRev. Everything is backed by functions.
1. Workflow Automations
Event-driven workflows that trigger on object changes — when a ticket is created, a conversation updated, an enhancement moved between stages. These are multi-step, Temporal-orchestrated workflows with retries, state management, and durable execution across process restarts.
2. Snap-Kit Invocations
Snap-kits are DevRev's developer extensibility layer. When a snap-in defines custom actions, UI handlers, or event listeners, the Function service is the runtime that executes them. A snap-kit function receives event context, runs in an isolated Lambda, and returns its result back through the same execution path as every other modality.
3. Keyring Verification
When an OAuth-backed connection needs token validation, the Function service runs the verification logic: checking token validity, refreshing credentials, and resolving keyring state. This is not a side path — it's the same function invocation infrastructure, just with a different trigger and execution context.
4. Snap-In Commands
Manual invocations triggered directly by users — a button click in the UI, an API trigger from an external system. The function executes synchronously, the result is returned in real-time. Same substrate, synchronous execution mode.
5. AirSync Extractors
Bidirectional data synchronization between DevRev and external systems. AirSync extractors run connector logic that maps, transforms, and syncs data across system boundaries. The Function service handles scheduling, retries, and result routing — the connector author just writes the extraction logic.
6. AI Agent Skills
When DevRev's AI agents execute a skill — looking up customer context, running a knowledge base search, classifying an object — the Function service handles execution. Each skill invocation carries agent session context: session ID, skill call ID, and the user the agent is acting on behalf of. The agent doesn't have a separate execution path. It uses the same validated, schema-aware operations that power production workflows.
---
Architecture: One Execution Stack
Every modality speaks the same language: operations. An operation has typed input ports, typed output ports, and an executor. The Function service routes execution regardless of whether the trigger was a webhook, an agent skill invocation, a cron job, or a user clicking a button.
┌─────────────────────────────────────────────────────┐
│ Modality Layer │
│ (Automations, Snap-Kits, Keyrings, Commands, ...) │
├─────────────────────────────────────────────────────┤
│ Operation Executor │
│ (Native Operations + Function Ops) │
├─────────────────────────────────────────────────────┤
│ Function Service │
│ (gRPC API, Routing, Auth, Logging) │
├─────────────────────────────────────────────────────┤
│ Code Runner │
│ (Lambda Invocation, Sandboxing, Limits) │
├─────────────────────────────────────────────────────┤
│ AWS Lambda (Production) │
│ Lambda RIE (Local Dev) │
└─────────────────────────────────────────────────────┘
Two Flavors of Operations
Native Operations — Implemented in Go, compiled into the service binary. These are control flow primitives: if_else, for_each, while, sleep, router. They execute in-process with near-zero overhead.
Function-Defined Operations — Implemented in TypeScript or Python, deployed as snap-in packages. These are the extensibility layer: custom actions, integrations, data transformations. They execute in isolated Lambda functions.
Both share the same interface:
type OperationExecutor interface {
ExecuteOperation(ctx, req) // Run the operation
ExecuteSchemaHandler(ctx, req) // Generate dynamic UI schema
ExecuteActivateHook(ctx, req) // Run on snap-in activation
ExecuteDeactivateHook(ctx, req) // Run on snap-in deactivation
}
The workflow engine doesn't know or care how an operation is implemented. A create_ticket step and an if_else step are identical from the orchestration layer's perspective.
---
Building and Deploying Functions
Before a function can execute, it goes through a structured build and deployment pipeline.
Step 1: Author
A developer writes TypeScript functions and registers them in a function-factory.ts:
import handler from './functions/on-ticket-created';
import verifyConnection from './functions/verify-connection';
export const functionFactory = {
on_ticket_created: handler,
verify_connection: verifyConnection,
} as const;
Each key in the factory maps to a function name that the Function service will route invocations to.
Step 2: Build
The snap-in is compiled via the DevRev CLI:
devrev snap_in_version create --path .
This transpiles TypeScript to JavaScript, bundles dependencies, and produces a zip artifact per function. The CLI also validates the manifest — checking that all referenced functions exist in the factory, that trigger schemas are valid, and that operation input/output types are declared.
Step 3: Upload
The zip artifacts are uploaded to the snap-in version registry. Each version is immutable — you can't modify a deployed version's code, only create a new version.
devrev snap_in_version package --path .
Step 4: Activate
A Temporal workflow orchestrates activation:
func ActivateSnapIn(ctx workflow.Context, param *SnapInActivateWorkflowArguments) error {
// Create event sources for all trigger functions
orgEventSourcesNameToIDMap, userEventSourceNames, err :=
createEventSources(ctx, svcDON, &snapInIDStr, ...)
// Execute activate hooks on each operation
for _, operation := range snapInVersion.Operations {
if operation.IsTrigger {
executeActivateHook(ctx, logger, ...)
}
}
}
Activation provisions everything the snap-in needs: event sources, service account, SQS queues for async delivery, IAM policies scoping what the snap-in can access. If any step fails, the workflow compensates — nothing is left in a half-activated state.
---
Running Code on Lambda
Once a function is deployed and activated, here's exactly what happens when it executes.
Invocation Path
Trigger fires (webhook / event / schedule / agent skill)
→ Function Service receives gRPC ExecuteOperation request
→ Resolves function: snap-in version → artifact → Lambda ARN
→ Injects execution context (tenant ID, actor, keyrings, input payload)
→ Invokes Lambda with structured payload
→ Receives response, classifies exit code
→ Returns structured result to caller
Payload Injection
User code doesn't touch the raw Lambda environment. It's wrapped in a security harness at invocation time:
func generateSafeUserCode(ctx context.Context, input *invokeCodeInputData) string {
inputsB64 := base64.StdEncoding.EncodeToString(inputJSON)
keyringsB64 := base64.StdEncoding.EncodeToString(keyringJSON)
userCodeB64 := base64.StdEncoding.EncodeToString(userCode)
replacer := strings.NewReplacer(
"__DEVREV_INVOKE_CODE_INPUTS_B64__", inputsB64,
"__DEVREV_INVOKE_CODE_KEYRINGS_B64__", keyringsB64,
"__DEVREV_INVOKE_CODE_USER_CODE_B64__", userCodeB64,
)
return replacer.Replace(pythonWrapper)
}
The harness sets up a restricted execution environment, decodes inputs, captures stdout/stderr within size limits, serializes output, and classifies exceptions — before handing control to user code.
Tenant Isolation
Each invocation carries a TenantId that provides isolation at the Lambda level. Different organizations' code never shares execution context:
lambdaInput := &lambda.InvokeInput{
FunctionName: aws.String(r.functionID),
TenantId: aws.String(namespace),
Payload: payloadBytes,
}
Local Development: Lambda RIE
For local development, the Function service targets Lambda RIE (Runtime Interface Emulator) instead of AWS Lambda. Lambda RIE runs as a local HTTP server that emulates the Lambda invoke API — same invocation path, same payload format, no AWS account required. Developers get identical execution semantics locally.
Exit Code Classification
The Function service distinguishes infrastructure failures from user code failures:
| Exit Code | Meaning | Error Type |
|-----------|---------|------------|
| 0 | Success | — |
| 1 | Runtime/syntax error in user code | InvalidArgument |
| 2 | Execution timeout | Aborted |
| 3 | Security violation (blocked import/operation) | InvalidArgument |
| 4 | Validation error (invalid code structure) | InvalidArgument |
Infrastructure failures (Lambda invoke errors, network timeouts) are surfaced as gRPC errors. User code failures are successful responses with non-zero exit codes. This distinction drives retry logic — you retry infrastructure failures, not user code bugs.
---
The Code Runner: Safe Execution for User Code
The Code Node operation — which lets workflow authors write inline Python — uses the same Lambda infrastructure but adds an additional security layer specific to running arbitrary user-supplied code. This is distinct from the standard function invocation path: most function invocations run pre-packaged TypeScript/Python that was already vetted at build time. The Code Runner adds runtime validation for the inline case.
AST validation runs before execution — inspecting the code structure to block dangerous imports, dunder attribute access (__class__, __bases__), and sandbox escape vectors.
Module blocklist prevents access to 72 Python modules covering filesystem access, process spawning, credential theft, and serialization vectors (pickle, marshal, shelve).
Restricted builtins lock down open, eval, __import__, input, breakpoint, and memoryview.
Resource limits:
Max Output Size: 512 KB
Max Memory: 256 MB
Default Timeout: 30 seconds
Max Execution: 3 minutes (workflow operations)
Max Serialization: 5 levels deep, 1000 items/list, 100 keys/dict
---
Temporal Orchestration: Reliability at Scale
For long-running workflows, the Function service is orchestrated by Temporal:
func getDefaultRetryPolicy() *temporal.RetryPolicy {
return &temporal.RetryPolicy{
InitialInterval: 5 * time.Second,
MaximumInterval: 60 * time.Second,
BackoffCoefficient: 2.0,
MaximumAttempts: 3,
}
}
Temporal provides automatic retries with exponential backoff, state persistence across process restarts, and event history with continue-as-new at 1,000 events to prevent unbounded growth.
For async operations (functions that take longer than a request-response cycle), an SQS-based callback processor handles completion signals. The function signals completion via SQS; the callback processor routes the result back to the waiting workflow activity.
---
The Operation Catalog
The Function service powers 180+ operations:
~100+ Trigger Operations:
- Object lifecycle:
account_created,ticket_updated,conversation_created - Scheduling:
timer_triggerfor cron-based execution - External:
api_triggerfor webhook invocations - AI:
ai_agent_skill_triggerfor agent skill execution - Sync:
airdrop_sync_run_startedfor connector orchestration
~80+ Action Operations:
- CRUD:
create_issue,update_ticket,get_account - AI:
ask_ai,classify_object,evaluate_sentiment - Search:
hybrid_search,agent_knowledge_fetch - Code:
execute_codewith streaming output (Code Node) - HTTP:
httpfor external API calls
Control Flow (Native Go):
if_else,for_each,while,routersleep_for,sleep_until,go_backset_variable,invoke_codetalk_to_agent,set_ai_agent_skill_output
Every operation — native or function-defined — shares the same schema system, the same validation pipeline, and the same execution semantics.
---
Why One Substrate Matters
When we added AI agent skills as a modality, we didn't build a new execution engine. We defined a new trigger type (ai_agent_skill_trigger), added agent context fields (session ID, skill call ID), and plugged it into the existing Function service. Security model, sandboxing, retry logic, monitoring — all inherited for free.
| | Separate Engines | Unified Substrate |
|---|---|---|
| Security model | N models to audit | 1 model to harden |
| Operation catalog | Duplicated per engine | Shared across all modalities |
| Schema validation | Inconsistent | Uniform |
| Monitoring | N dashboards | 1 observability stack |
| New modality cost | Build from scratch | Plug into existing substrate |
Every security fix hardens all six modalities. Every performance optimization speeds up all six. Every new capability is available to all six.
One substrate. Every execution context. That's the architecture.
---
Production Numbers
- gRPC message limits: 40 MB receive, 24 MB send
- Temporal event buffer: 10,000 events per workflow
- Continue-as-new threshold: 1,000 events
- Default retry policy: 3 attempts, exponential backoff (5s → 60s)
- Function timeout: 30 seconds default, 3 minutes max
- Async callback: SQS-based with dedicated worker fleet
