OCI Monitoring and Observability - Expert Knowledge
🏗️ Use OCI Landing Zone Terraform Modules
Don't reinvent the wheel. Use oracle-terraform-modules/landing-zone for observability stack.
Landing Zone solves:
- ❌ Bad Practice #10: No logging, monitoring, notifications (Landing Zone deploys complete observability)
- ❌ Bad Practice #7: Limited security services (Landing Zone integrates Cloud Guard, VSS, OSMS)
This skill provides: Metrics, alarms, and troubleshooting for monitoring deployed WITHIN a Landing Zone.
⚠️ OCI CLI/API Knowledge Gap
You don't know OCI CLI commands or OCI API structure.
Your training data has limited and outdated knowledge of:
- OCI CLI syntax and parameters (updates monthly)
- OCI API endpoints and request/response formats
- Monitoring service CLI operations (
oci monitoring alarm,oci monitoring metric) - Metric namespaces and MQL (Monitoring Query Language)
- Latest Logging and Service Connector features
When OCI operations are needed:
- Use exact CLI commands from this skill's references
- Do NOT guess metric namespace names
- Do NOT assume AWS CloudWatch patterns work in OCI
- Load reference files for detailed MQL documentation
What you DO know:
- General observability concepts
- Alerting and threshold design principles
- Log aggregation patterns
This skill bridges the gap by providing current OCI-specific monitoring patterns and gotchas.
NEVER Do This
❌ NEVER assume metrics are instant (10-15 minute lag)
- Metrics published every 1-5 minutes
- Processing delay: 5-10 minutes
- Total lag: 10-15 minutes from event to visible metric
- Don't debug "missing metrics" within first 15 minutes of resource creation
❌ NEVER use = for alarm thresholds with sparse metrics
# WRONG - alarm never fires if metric has gaps
MetricName[1m].mean() = 0
# RIGHT - handle missing data
MetricName[1m]{dataMissing=zero}.mean() > 0
❌ NEVER forget metric dimensions (causes "no data")
# WRONG - missing required dimension
CPUUtilization[1m].mean()
# RIGHT - include resourceId dimension
CPUUtilization[1m]{resourceId="<instance-ocid>"}.mean()
❌ NEVER set alarm thresholds without trigger delay (alert fatigue)
# BAD - fires on every CPU spike
CPUUtilization[1m].mean() > 80
# BETTER - sustained high CPU
CPUUtilization[5m].mean() > 80
Trigger delay: 5 minutes (fires after 5 consecutive breaches)
❌ NEVER create alarms without notification channels
# WRONG - alarm fires but nobody knows
oci monitoring alarm create ... --destinations '[]'
# RIGHT - always link to notification topic
oci monitoring alarm create ... --destinations '["<notification-topic-ocid>"]'
Cost impact: Undetected outages cost $5,000-50,000/hour in production
❌ NEVER ignore Cloud Guard findings (security audit failure)
- Cloud Guard detects misconfigurations BEFORE they become incidents
- Integrate Cloud Guard → Notifications → Email/Slack/PagerDuty
- Cost impact: $100,000+ per security breach vs $0 for proactive remediation
Metric Namespace Gotchas
OCI Metrics Use Service-Specific Namespaces:
| Service | Namespace | Example Metric |
|---|---|---|
| Compute | oci_computeagent | CPUUtilization, MemoryUtilization |
| Autonomous DB | oci_autonomous_database | CpuUtilization, StorageUtilization |
| Load Balancer | oci_lbaas | HttpRequests, UnHealthyBackendServers |
| Object Storage | oci_objectstorage | ObjectCount, BytesUploaded |
Common Mistake: Using wrong namespace (oci_compute vs oci_computeagent)
Alarm Missing Data Handling
| Setting | Behavior | Use When |
|---|---|---|
treatMissingDataAsBreaching | Alarm fires if no data | Critical services (outage = breach) |
treatMissingDataAsNotBreaching | Alarm silent if no data | Optional monitoring |
{dataMissing=zero} | Treat missing as 0 | Counters (requests/sec) |
Log Collection Common Gaps
Problem: Logs not showing in Log Analytics
Logs not appearing?
├─ Is log enabled on resource?
│ └─ Compute: oci-compute-agent must be running
│ └─ Function: Logging enabled in function config
│
├─ Is Service Connector configured?
│ └─ Source: Log Group → Target: Log Analytics
│ └─ Check: Service Connector status = ACTIVE
│
├─ IAM policy for Service Connector?
│ └─ "Allow any-user to use log-content in tenancy"
│ └─ "Allow service loganalytics to READ logcontent in tenancy"
│
└─ 10-15 minute ingestion lag?
└─ Wait before debugging
Metric Query Optimization
Expensive (slow):
# Queries ALL instances
CPUUtilization[1m].mean()
Optimized (filter by dimension):
# Query specific instance
CPUUtilization[1m]{resourceId='<instance-ocid>'}.mean()
Cost: Queries free, but rate limited (1000 req/min)
Progressive Loading References
OCI Monitoring Reference (Official Oracle Documentation)
WHEN TO LOAD oci-monitoring-reference.md:
- Need comprehensive list of all OCI service metrics
- Understanding MQL (Monitoring Query Language) in depth
- Implementing complex alarm conditions and composites
- Need official Oracle guidance on Logging and Service Connector
- Setting up Log Analytics and APM integration
Do NOT load for:
- Quick alarm setup (examples in this skill)
- Common metric patterns (tables above)
- Troubleshooting decision trees (covered above)
When to Use This Skill
- Alarms: threshold configuration, missing data handling, trigger delay
- Troubleshooting: metrics not showing, alarms not firing, namespace errors
- Log collection: Service Connector, IAM policies, missing logs
- Performance: query optimization, dimension filtering