Type4Me macOS Voice Input
Skill by ara.so — Daily 2026 Skills collection.
Type4Me is a macOS voice input tool that captures audio via global hotkey, transcribes it using local (SherpaOnnx/Paraformer/Zipformer) or cloud (Volcengine/Deepgram) ASR engines, optionally post-processes text via LLM, and injects the result into any app. All credentials and history are stored locally — no telemetry, no cloud sync.
Architecture Overview
Type4Me/
├── ASR/ # ASR engine abstraction
│ ├── ASRProvider.swift # Provider enum + protocols
│ ├── ASRProviderRegistry.swift # Plugin registry
│ ├── Providers/ # Per-vendor config files
│ ├── SherpaASRClient.swift # Local streaming ASR
│ ├── SherpaOfflineASRClient.swift
│ ├── VolcASRClient.swift # Volcengine streaming ASR
│ └── DeepgramASRClient.swift # Deepgram streaming ASR
├── Bridge/ # SherpaOnnx C API Swift bridge
├── Audio/ # Audio capture
├── Session/ # Core state machine: record→ASR→inject
├── Input/ # Global hotkey management
├── Services/ # Credentials, hotwords, model manager
├── Protocol/ # Volcengine WebSocket codec
└── UI/ # SwiftUI (FloatingBar + Settings)
Installation
Prerequisites
# Xcode Command Line Tools
xcode-select --install
# CMake (for local ASR engine)
brew install cmake
Build & Deploy from Source
git clone https://github.com/joewongjc/type4me.git
cd type4me
# Step 1: Compile SherpaOnnx local engine (~5 min, one-time)
bash scripts/build-sherpa.sh
# Step 2: Build, bundle, sign, install to /Applications, and launch
bash scripts/deploy.sh
Download Pre-built App
Download Type4Me-v1.2.3.dmg from releases (cloud ASR only, no local engine):
https://github.com/joewongjc/type4me/releases/tag/v1.2.3
If macOS blocks the app:
xattr -d com.apple.quarantine /Applications/Type4Me.app
Download Local ASR Models
mkdir -p ~/Library/Application\ Support/Type4Me/Models
# Option A: Lightweight ~20MB
tar xjf ~/Downloads/sherpa-onnx-streaming-zipformer-small-ctc-zh-int8-2025-04-01.tar.bz2 \
-C ~/Library/Application\ Support/Type4Me/Models/
# Option B: Balanced ~236MB (recommended)
tar xjf ~/Downloads/sherpa-onnx-streaming-zipformer-ctc-multi-zh-hans-2023-12-13.tar.bz2 \
-C ~/Library/Application\ Support/Type4Me/Models/
# Option C: Bilingual Chinese+English ~1GB
tar xjf ~/Downloads/sherpa-onnx-streaming-paraformer-bilingual-zh-en.tar.bz2 \
-C ~/Library/Application\ Support/Type4Me/Models/
Expected structure for Paraformer model:
~/Library/Application Support/Type4Me/Models/
└── sherpa-onnx-streaming-paraformer-bilingual-zh-en/
├── encoder.int8.onnx
├── decoder.int8.onnx
└── tokens.txt
Key Protocols
SpeechRecognizer Protocol
Every ASR client must implement this protocol:
protocol SpeechRecognizer: AnyObject {
/// Start a new recognition session
func startRecognition() async throws
/// Feed raw PCM audio data
func appendAudio(_ buffer: AVAudioPCMBuffer) async
/// Stop and get final result
func stopRecognition() async throws -> String
/// Cancel without result
func cancelRecognition() async
/// Streaming partial results (optional)
var partialResultHandler: ((String) -> Void)? { get set }
}
ASRProviderConfig Protocol
Each vendor's credential definition:
protocol ASRProviderConfig {
/// Unique identifier string
static var providerID: String { get }
/// Display name in Settings UI
static var displayName: String { get }
/// Credential fields shown in Settings
static var credentialFields: [CredentialField] { get }
/// Validate credentials before use
static func validate(_ credentials: [String: String]) -> Bool
/// Create the recognizer instance
static func createClient(
credentials: [String: String],
config: RecognitionConfig
) throws -> SpeechRecognizer
}
Adding a New ASR Provider
Step 1: Create Provider Config
Create Type4Me/ASR/Providers/OpenAIWhisperProvider.swift:
import Foundation
struct OpenAIWhisperProvider: ASRProviderConfig {
static let providerID = "openai_whisper"
static let displayName = "OpenAI Whisper"
static let credentialFields: [CredentialField] = [
CredentialField(
key: "api_key",
label: "API Key",
placeholder: "sk-...",
isSecret: true
),
CredentialField(
key: "model",
label: "Model",
placeholder: "whisper-1",
isSecret: false
)
]
static func validate(_ credentials: [String: String]) -> Bool {
guard let apiKey = credentials["api_key"], !apiKey.isEmpty else {
return false
}
return apiKey.hasPrefix("sk-")
}
static func createClient(
credentials: [String: String],
config: RecognitionConfig
) throws -> SpeechRecognizer {
guard let apiKey = credentials["api_key"] else {
throw ASRError.missingCredential("api_key")
}
let model = credentials["model"] ?? "whisper-1"
return OpenAIWhisperASRClient(apiKey: apiKey, model: model, config: config)
}
}
Step 2: Implement the ASR Client
Create Type4Me/ASR/OpenAIWhisperASRClient.swift:
import Foundation
import AVFoundation
final class OpenAIWhisperASRClient: SpeechRecognizer {
var partialResultHandler: ((String) -> Void)?
private let apiKey: String
private let model: String
private let config: RecognitionConfig
private var audioData: Data = Data()
init(apiKey: String, model: String, config: RecognitionConfig) {
self.apiKey = apiKey
self.model = model
self.config = config
}
func startRecognition() async throws {
audioData = Data()
}
func appendAudio(_ buffer: AVAudioPCMBuffer) async {
// Convert PCM buffer to raw bytes and accumulate
guard let channelData = buffer.floatChannelData?[0] else { return }
let frameCount = Int(buffer.frameLength)
let bytes = UnsafeBufferPointer(start: channelData, count: frameCount)
// Convert Float32 PCM to Int16 for Whisper API
let int16Samples = bytes.map { sample -> Int16 in
return Int16(max(-32768, min(32767, Int(sample * 32767))))
}
int16Samples.withUnsafeBytes { ptr in
audioData.append(contentsOf: ptr)
}
}
func stopRecognition() async throws -> String {
// Build multipart form request to Whisper API
var request = URLRequest(url: URL(string: "https://api.openai.com/v1/audio/transcriptions")!)
request.httpMethod = "POST"
request.setValue("Bearer \(apiKey)", forHTTPHeaderField: "Authorization")
let boundary = UUID().uuidString
request.setValue("multipart/form-data; boundary=\(boundary)",
forHTTPHeaderField: "Content-Type")
var body = Data()
// Append audio file part
body.append("--\(boundary)\r\n".data(using: .utf8)!)
body.append("Content-Disposition: form-data; name=\"file\"; filename=\"audio.raw\"\r\n".data(using: .utf8)!)
body.append("Content-Type: audio/raw\r\n\r\n".data(using: .utf8)!)
body.append(audioData)
body.append("\r\n".data(using: .utf8)!)
// Append model part
body.append("--\(boundary)\r\n".data(using: .utf8)!)
body.append("Content-Disposition: form-data; name=\"model\"\r\n\r\n".data(using: .utf8)!)
body.append("\(model)\r\n".data(using: .utf8)!)
body.append("--\(boundary)--\r\n".data(using: .utf8)!)
request.httpBody = body
let (data, response) = try await URLSession.shared.data(for: request)
guard let httpResponse = response as? HTTPURLResponse,
httpResponse.statusCode == 200 else {
throw ASRError.networkError("Whisper API returned error")
}
let result = try JSONDecoder().decode(WhisperResponse.self, from: data)
return result.text
}
func cancelRecognition() async {
audioData = Data()
}
}
private struct WhisperResponse: Codable {
let text: String
}
Step 3: Register the Provider
In Type4Me/ASR/ASRProviderRegistry.swift, add to the all array:
struct ASRProviderRegistry {
static let all: [any ASRProviderConfig.Type] = [
SherpaParaformerProvider.self,
VolcengineProvider.self,
DeepgramProvider.self,
OpenAIWhisperProvider.self, // ← Add your provider here
]
}
Credentials Storage
Credentials are stored at ~/Library/Application Support/Type4Me/credentials.json with permissions 0600. Never hardcode secrets — always load via CredentialStore:
// Reading credentials
let store = CredentialStore.shared
let apiKey = store.get(providerID: "openai_whisper", key: "api_key")
// Writing credentials
store.set(providerID: "openai_whisper", key: "api_key", value: userInputKey)
// Checking if configured
let isConfigured = store.isConfigured(providerID: "openai_whisper",
fields: OpenAIWhisperProvider.credentialFields)
Custom Processing Modes with Prompt Variables
Processing modes use LLM post-processing with three context variables:
| Variable | Value |
|---|---|
{text} | Recognized speech text |
{selected} | Text selected in active app at record start |
{clipboard} | Clipboard content at record start |
Example custom mode prompts:
// Translate selection using voice command
let translatePrompt = """
The user selected this text: {selected}
Voice command: {text}
Execute the command on the selected text. Output only the result.
"""
// Code review via voice
let codeReviewPrompt = """
Code to review:
{clipboard}
Review instruction: {text}
Provide focused feedback addressing the instruction.
"""
// Email reply drafting
let emailPrompt = """
Original email: {selected}
My reply intent (spoken): {text}
Write a professional email reply. Output only the email body.
"""
Built-in Processing Modes
enum ProcessingMode {
case fast // Direct ASR output, zero latency
case performance // Dual-channel: streaming + offline refinement
case englishTranslation // Chinese speech → English text
case promptOptimize // Raw prompt → optimized prompt via LLM
case command // Voice command + selected/clipboard context → LLM action
case custom(prompt: String) // User-defined prompt template
}
Session State Machine
The core recording flow in Session/:
[Idle]
→ hotkey pressed → [Recording] → audio streams to ASR client
→ hotkey released/pressed again → [Processing]
→ ASR returns text → [LLM Post-processing] (if mode requires)
→ [Injecting] → text injected into active app
→ [Idle]
Updating After Source Changes
cd type4me
git pull
bash scripts/deploy.sh
# SherpaOnnx does NOT need recompiling unless engine version changed
Troubleshooting
App won't open (security warning)
xattr -d com.apple.quarantine /Applications/Type4Me.app
Local model not recognized in Settings
Verify the directory structure exactly matches:
ls ~/Library/Application\ Support/Type4Me/Models/sherpa-onnx-streaming-paraformer-bilingual-zh-en/
# Must show: encoder.int8.onnx decoder.int8.onnx tokens.txt
SherpaOnnx build fails
# Ensure cmake is installed
brew install cmake
# Clean and retry
rm -rf Frameworks/
bash scripts/build-sherpa.sh
New ASR provider not appearing in Settings
- Confirm the provider type is added to
ASRProviderRegistry.all - Ensure
providerIDis unique across all providers - Clean build:
swift package clean && bash scripts/deploy.sh
Audio not captured / no floating bar
- Grant microphone permission: System Settings → Privacy & Security → Microphone → Type4Me ✓
- Grant Accessibility permission for text injection: System Settings → Privacy & Security → Accessibility → Type4Me ✓
Credentials not saving
# Check file exists and has correct permissions
ls -la ~/Library/Application\ Support/Type4Me/credentials.json
# Should show: -rw------- (0600)
# Fix permissions if needed:
chmod 0600 ~/Library/Application\ Support/Type4Me/credentials.json
Export history to CSV
Open Settings → History → select date range → Export CSV. The SQLite database is at:
~/Library/Application\ Support/Type4Me/history.db
# Direct query:
sqlite3 ~/Library/Application\ Support/Type4Me/history.db \
"SELECT datetime(timestamp,'unixepoch'), text FROM records ORDER BY timestamp DESC LIMIT 20;"
System Requirements
- macOS 14.0 (Sonoma) or later
- Apple Silicon (M1/M2/M3/M4) recommended for local ASR inference
- Xcode Command Line Tools + CMake for source builds
- Internet connection only needed for cloud ASR providers