Hansei (反省; "self-reflection") is a central idea in Japanese culture, meaning to acknowledge one’s own mistake and to pledge improvement.
A tokio post-mortem debugger.
hansei is a tool to observe the state of the tokio runtime and its associated async tasks in
core dumps.
Core dumps generated on illumos and Linux are supported. Analysis of core dumps can be performed on
any platform hansei can build on, e.g., an illumos core can be debugged on macOS.
The binary that dumped core must use v0 symbol mangling, which is the default since Rust 1.97.0+,
though older releases of Rust support it with the -C symbol-mangling-version=v0 flag.
Both the current-thread and multi-thread executors are supported, as are programs that use
multiple runtimes and the tokio_unstable feature.
Using the hansei tokio-info extract subcommand, hansei can create a tokio-info (.tinfo) file
that contains only the small subset of struct layouts from the DWARF hansei needs. The tokio-info
format is vastly smaller, and therefore more convenient to transfer than a release build with full
debug information:
$ ls -lh cores/nexus*
-rw-r--r--@ 1 wfc 2.3G Aug 22 14:18 nexus.core
-rwxr-xr-x@ 1 wfc 4.1G Aug 22 14:44 nexus.debug_bin*
-rw-r--r--@ 1 wfc 9.3M Sep 2 22:32 nexus.tinfoBuild hansei with cargo build --release --bin hansei.
hansei always operates against core dumps; live processes are not supported. You will need debug
information for the process that dumped core. This could be the original binary if it was built
with debug=true, it could be a split debug info .dwp file on Linux, or it could be a separate
build of the original binary with debug=true set. If you are using a separate build, it is
critical that all dependencies and build flags (including the profile, e.g., release) be the same.
Otherwise the struct layouts hansei extracts from the DWARF may vary from the layout used in the
core, invalidating its analysis. hansei will attempt to validate that the DWARF matches the core,
but this is best-effort.
You can directly invoke hansei against the core using:
hansei --core mybin.core --debug-info mybin.dbg
To analyze a Linux core, the original binary must also be present so we can read its symbol table. This is not required for illumos cores, as the symbol table is included as part of the core on that platform.
hansei --core mybin.core --debug-info mybin.dbg --binary mybin
This will immediately process the DWARF and launch a hansei session. DWARF processing is
computationally expensive, and can typically take tens of seconds on larger binaries. To avoid
needing to do this each time for a binary, the extracted tokio-info can be saved from within the
hansei REPL using the save-tokio-info command.
To pre-generate the tokio-info, use the tokio-info extract subcommand:
hansei tokio-info extract mybin.dbg --output mybin.tinfo
This file then replaces --debug-info and makes startup significantly faster.
hansei --core mybin.core --tokio-info mybin.tinfo
Non-interactive sessions can be started with the --exec flag, or by passing commands in via
stdin. Commands can be separated with semicolons.
hansei --core mybin.core --tokio-info mybin.tinfo --exec 'census; tasks --with state running' # Equivalent echo 'census; tasks --with state running' | hansei --core mybin.core --tokio-info mybin.tinfo
All commands can be tab completed, as can most arguments to commands. Run help for a list of all
commands, and help <COMMAND> for details on a given command.
The census command gives a brief summary of the threads, tasks, and futures present in the
tokio runtime.
hansei> census
Threads: 33 lwps, 33 in runtime 0 @ 0x5629d639a9a0
COUNT RT ROLE STATE LWPS
31 0 worker parked 2040416, 2040429, 2040432, …
1 0 worker polling 2040415 (task 35)
1 0 entered runtime block_on caller 2040413
0 0 blocking pool 0 idle, 0 busy —
[33 lwps]
Tasks: 3 owned by runtime 0 @ 0x5629d639a9a0
COUNT STATE
2 idle
1 running
[3 tasks]
COUNT TYPE / WAITING ON
1 async block rama_core::rt::executor::Executor::spawn_task<tracing::instrument::Instrumented<rama_tcp::server::listener::TcpListener::serve::{async_fn#0}::{async_block_env#3}<rama_http_types::body_limit_layer::BodyLimitS…
└─ 1 — (mid-poll)
1 async block tokio_graceful::guard::ShutdownGuard::spawn_task_fn<http_mitm_proxy_boring::main::{async_block#0}::{closure_env#0}, http_mitm_proxy_boring::main::{async_block#0}::{closure#0}>
└─ 1 io
1 async block tokio_graceful::shutdown::ShutdownBuilder::build<tokio_graceful::shutdown::default_signal>
└─ 1 future core::future::poll_fn::PollFn<tokio_graceful::shutdown::default_signal::{async_fn#0}::{closure_env#1}>
[3 types]
Futures: 23 in flight, on 60 await-chain frames, up to 8 deep
COUNT HELD IN
3 task (its own await chain)
20 frame (off any await chain)
0 set (0 FuturesUnordered)
[23 futures]
COUNT TYPE / WAITING ON
5 future tokio_graceful::trigger::Receiver
2 future Pin<&mut futures_util::future::future::fuse::Fuse<tokio_graceful::guard::ShutdownGuard::cancelled>>
2 future Pin<&mut rama_http_core::server::conn::auto::UpgradeableConnection<Pin<Box<rama_tcp::stream::TcpStream>>, rama_http_core::service::RamaHttpService<Arc<rama_http::layer::trace::service::Trace<rama_core::layer::con…
2 future Pin<&mut rama_tcp::server::listener::TcpListener::serve::{async_fn#0}<rama_http_types::body_limit_layer::BodyLimitService<rama_http_backend::server::service::HttpService<rama_http_core::server::conn::auto::Builde…
1 async block rama_http_core::proto::h1::dispatch::Server::recv_msg<rama_http_core::service::RamaHttpService<Arc<rama_http::layer::trace::service::Trace<rama_core::layer::consume_err::ConsumeErr<rama_http::layer::proxy_au…
8 (8 more types)
[13 types, 5 shown]
There are three object types that can be reviewed in more depth: tasks, futures, and threads. Each
has a plural command (tasks) that can be used to list, group, and filter objects, and a singular
command (task) that inspects one. This syntax follows the general pattern the excellent
Delve debugger for Go uses.
Counts by a field can be found using --group.
hansei> tasks --group state
COUNT STATE TASKS
22495 idle 129, 130, 131, …
3 running 2436, 2439, 2445
1 blocking 765
[3 groups]
Use --with and --without to filter results.
hansei> tasks --with state running --without id 2445
ID STATE AWAITING AT WAITING ON FUTURE
2436 running parallel-task-set/src/lib.rs:97 — (mid-poll on lwp 115) async block parallel_task_set::ParallelTaskSet::spawn::{async_fn#0}<support_bundle_collection::step::CompletedCollectionStep, support_bundle_collection…
2439 running parallel-task-set/src/lib.rs:97 — (mid-poll on lwp 124) async block parallel_task_set::ParallelTaskSet::spawn::{async_fn#0}<support_bundle_collection::step::CompletedCollectionStep, support_bundle_collection…
[2 tasks]
The --exec flag can execute single-object commands against all objects returned:
hansei> tasks --with state running --without id 2445 --exec task
task 2436
state: running
thread: 115
type: async block parallel_task_set::ParallelTaskSet::spawn::{async_fn#0}<support_bundle_collection::step::CompletedCollectionStep, support_bundle_collection::collection::BundleCollection::run_collect_bundle_steps::{async_fn#0}>
awaiting at: parallel-task-set/src/lib.rs:97
spawned at: parallel-task-set/src/lib.rs:96:18
defined at: parallel-task-set/src/lib.rs:96
held futures: 0
join sets: 2 (7057 futures)
task 2439
state: running
thread: 124
type: async block parallel_task_set::ParallelTaskSet::spawn::{async_fn#0}<support_bundle_collection::step::CompletedCollectionStep, support_bundle_collection::collection::BundleCollection::run_collect_bundle_steps::{async_fn#0}>
awaiting at: parallel-task-set/src/lib.rs:97
spawned at: parallel-task-set/src/lib.rs:96:18
defined at: parallel-task-set/src/lib.rs:96
held futures: 0
join sets: 2 (7126 futures)
[Executed against 2 tasks, 0 failed]
Use the singular form of a command to set the session cursor to a particular object. The prompt will show the cursor’s location when set.
hansei> task 2439
task 2439
state: running
thread: 124
type: async block parallel_task_set::ParallelTaskSet::spawn::{async_fn#0}<support_bundle_collection::step::CompletedCollectionStep, support_bundle_collection::collection::BundleCollection::run_collect_bundle_steps::{async_fn#0}>
awaiting at: parallel-task-set/src/lib.rs:97
spawned at: parallel-task-set/src/lib.rs:96:18
defined at: parallel-task-set/src/lib.rs:96
held futures: 0
join sets: 2 (7126 futures)
hansei : task 2439 #0>
A command appended to task and other singular commands will execute against that object without
moving the cursor.
hansei> task 2439 trace
warning: task 2439 is running on lwp 124; its state may be torn
#0 future futures_util::stream::futures_unordered::FuturesUnordered<support_bundle_collection::steps::host_info::save_zone_log_zip_or_error>
(<no_state>, 3 locals)
#1 async fn support_bundle_collection::steps::host_info::collect_data_from_sled
awaiting at support-bundle-collection/src/steps/host_info.rs:250 (Suspend6, 10 locals)
#2 async block support_bundle_collection::steps::host_info::spawn_query_all_sleds::{async_fn#0}::{closure#0} [dyn]
awaiting at support-bundle-collection/src/steps/host_info.rs:58 (Suspend0, 0 locals)
#3 async fn support_bundle_collection::step::CollectionStep::run
awaiting at support-bundle-collection/src/step.rs:55 (Suspend0, 3 locals)
#4 async block support_bundle_collection::collection::BundleCollection::run_collect_bundle_steps::{async_fn#0}
awaiting at support-bundle-collection/src/collection.rs:204 (Suspend0, 0 locals)
#5 async block parallel_task_set::ParallelTaskSet::spawn::{async_fn#0}<support_bundle_collection::step::CompletedCollectionStep, support_bundle_collection::collection::BundleCollection::run_collect_bundle_steps::{async_fn…
awaiting at parallel-task-set/src/lib.rs:97 (Suspend0, 0 locals)
For tasks and futures, an async backtrace can be printed using the trace command. This is heavily
inspired by the lilos debugger described in https://cliffle.com/blog/lildb.
For tasks that are running on a thread, the --native flag will combine the stack trace of the
thread with the async trace, showing only the native frames between the task’s poll function and
the top of the stack.
hansei : task 2439 #0> trace --native
warning: task 2439 is running on lwp 124; its state may be torn
mid-poll on lwp 124
0xfffff5ffef0a7a34 __lwp_unpark
0xfffff5ffef1e2d4b vmem_xalloc
0xfffff5ffef1db79a memalign
0xfffff5ffef03c318 posix_memalign
0x000000000e350965 __rustc::__rdl_alloc
0x000000000d782cee <tokio::runtime::scheduler::multi_thread::handle::Handle>::bind_new_task::<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<dyn core::future::future::Future<Output = ()> + core::marker::…
0x000000000d77c5ab tokio::task::spawn::spawn::<tracing::instrument::Instrumented<core::pin::Pin<alloc::boxed::Box<dyn core::future::future::Future<Output = ()> + core::marker::Send>>>>
0x000000000d738866 <hyper_util::rt::tokio::TokioExecutor as hyper::rt::Executor<core::pin::Pin<alloc::boxed::Box<dyn core::future::future::Future<Output = ()> + core::marker::Send>>>>::execute
0x000000000d776dd2 <core::pin::Pin<alloc::boxed::Box<<hyper_util::client::legacy::client::Client<reqwest::connect::Connector, reqwest::async_impl::body::Body>>::connect_to::{closure#0}::{closure#1}::{closure#0}>> as core::f…
0x000000000d774e0c <futures_util::future::try_future::try_flatten::TryFlatten<futures_util::future::try_future::MapOk<futures_util::future::try_future::MapErr<hyper_util::service::oneshot::Oneshot<reqwest::connect::Connecto…
0x000000000d704c5b <hyper_util::common::lazy::Lazy<<hyper_util::client::legacy::client::Client<reqwest::connect::Connector, reqwest::async_impl::body::Body>>::connect_to::{closure#0}, futures_util::future::either::Either<fu…
0x000000000d6fce54 <hyper_util::client::legacy::client::Client<reqwest::connect::Connector, reqwest::async_impl::body::Body>>::send_request::{closure#0}
0x000000000d6cf766 <reqwest::async_impl::client::HyperService as tower_service::Service<http::request::Request<reqwest::async_impl::body::Body>>>::call::{closure#0}
0x000000000d6e66e8 <tower::retry::future::ResponseFuture<reqwest::retry::Policy, reqwest::async_impl::client::HyperService, http::request::Request<reqwest::async_impl::body::Body>> as core::future::future::Future>::poll
0x000000000d6d7bdb <reqwest::cookie::service::ResponseFuture<tower::retry::Retry<reqwest::retry::Policy, reqwest::async_impl::client::HyperService>, reqwest::async_impl::body::Body> as core::future::future::Future>::poll
0x000000000d6e4634 <tower_http::follow_redirect::ResponseFuture<reqwest::cookie::service::CookieService<tower::retry::Retry<reqwest::retry::Policy, reqwest::async_impl::client::HyperService>>, reqwest::async_impl::body::Bod…
0x000000000d6db94f <reqwest::async_impl::client::PendingRequest as core::future::future::Future>::poll
0x000000000d6db8cd <reqwest::async_impl::client::Pending as core::future::future::Future>::poll
0x000000000af7ae21 <core::future::poll_fn::PollFn<support_bundle_collection::steps::host_info::save_zone_log_zip_or_error::{closure#0}::{closure#5}> as core::future::future::Future>::poll
0x000000000b020b51 <futures_util::stream::futures_unordered::FuturesUnordered<support_bundle_collection::steps::host_info::save_zone_log_zip_or_error::{closure#0}> as futures_util::stream::stream::StreamExt>::poll_next_unpin
#0 future futures_util::stream::futures_unordered::FuturesUnordered<support_bundle_collection::steps::host_info::save_zone_log_zip_or_error>
(<no_state>, 3 locals)
#1 async fn support_bundle_collection::steps::host_info::collect_data_from_sled
awaiting at support-bundle-collection/src/steps/host_info.rs:250 (Suspend6, 10 locals)
#2 async block support_bundle_collection::steps::host_info::spawn_query_all_sleds::{async_fn#0}::{closure#0} [dyn]
awaiting at support-bundle-collection/src/steps/host_info.rs:58 (Suspend0, 0 locals)
#3 async fn support_bundle_collection::step::CollectionStep::run
awaiting at support-bundle-collection/src/step.rs:55 (Suspend0, 3 locals)
#4 async block support_bundle_collection::collection::BundleCollection::run_collect_bundle_steps::{async_fn#0}
awaiting at support-bundle-collection/src/collection.rs:204 (Suspend0, 0 locals)
#5 async block parallel_task_set::ParallelTaskSet::spawn::{async_fn#0}<support_bundle_collection::step::CompletedCollectionStep, support_bundle_collection::collection::BundleCollection::run_collect_bundle_steps::{async_fn…
awaiting at parallel-task-set/src/lib.rs:97 (Suspend0, 0 locals)
Navigate between async frames using the up, down, and frame commands. The current frame number
is displayed in the prompt.
A given async frame may contain a number of local variables that are live at the frame’s current
suspend point. The locals command will list them.
hansei : task 2439 #0> locals
ready_to_run_queue: alloc::sync::Arc<futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<support_bundle_collection::steps::host_info::save_zone_log_zip_or_error::{async_fn_env#0}>, alloc::alloc::Global> = 0x4d0a6650 -> alloc::sync::ArcInner<futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<support_bundle_collection::steps::host_info::save_zone_log_zip_or_error::{async_fn_env#0}>> {
strong: 1,
weak: 7128,
data: futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<support_bundle_collection::steps::host_info::save_zone_log_zip_or_error::{async_fn_env#0}> { .. } @ 0x4d0a6660,
}
head_all: core::sync::atomic::Atomic<*mut futures_util::stream::futures_unordered::task::Task<support_bundle_collection::steps::host_info::save_zone_log_zip_or_error::{async_fn_env#0}>> = 0x4e007960 (task 2439 via FuturesUnordered) -> futures_util::stream::futures_unordered::task::Task<support_bundle_collection::steps::host_info::save_zone_log_zip_or_error::{async_fn_env#0}> {
future: core::cell::UnsafeCell<core::option::Option<support_bundle_collection::steps::host_info::save_zone_log_zip_or_error::{async_fn_env#0}>> { .. } @ 0x4e007968 (task 2439 via FuturesUnordered),
next_all: task 2439 via FuturesUnordered,
prev_all: null,
len_all: 7126,
next_ready_to_run: null,
ready_to_run_queue: alloc::sync::Weak<futures_util::stream::futures_unordered::ready_to_run_queue::ReadyToRunQueue<support_bundle_collection::steps::host_info::save_zone_log_zip_or_error::{async_fn_env#0}>, alloc::alloc::Global> { .. } @ 0x4e007960 (task 2439 via FuturesUnordered),
queued: core::sync::atomic::Atomic<bool> { .. } @ 0x4e007f18 (task 2439 via FuturesUnordered),
woken: core::sync::atomic::Atomic<bool> { .. } @ 0x4e007f19 (task 2439 via FuturesUnordered),
}
is_terminated: core::sync::atomic::Atomic<bool> = 0
The print command will display the details of a specific local, or a field in within it. Press
Tab to receive autocomplete suggestions for field names.
hansei : task 2439 #0> print head_all.future
core::option::Option<support_bundle_collection::steps::host_info::save_zone_log_zip_or_error::{async_fn_env#0}> = Some(3 {
logger: 0x111e0a40 -> slog::Logger<alloc::sync::Arc<dyn slog::SendSyncRefUnwindSafeDrain<Ok=(), Err=core::convert::Infallible>, alloc::alloc::Global>> { .. },
zone: "oxz_propolis-server_bb36cd0f-4603-425c-9a58-5f17bc454b85",
path: "/var/tmp/.tmpXWNr2l/rack/de608e01-b8e4-4d93-b972-a7dbed36dd22/sled/a2adea92-b56e-44fc-8a0d-7d63b5fd3b93",
disabled: 0,
futures: (tokio_util::sync::cancellation_token::WaitForCancellationFuture, sled_agent_client::{impl#11}::support_logs_download::{async_fn_env#0}) (..) @ 0x4e007b40,
__awaitee: core::future::poll_fn::PollFn<support_bundle_collection::steps::host_info::save_zone_log_zip_or_error::{async_fn#0}::{closure_env#5}> { .. } @ 0x4e007ee8,
})
How many levels the printer will recurse into an object is controlled by the depth option in the
config command. Certain known types, e.g., Vec, String, will be printed in a human-friendly
format, rather than the raw struct fields. The ugly setting disables this behavior.
When the await chain bottoms out in a recognized wait primitive, a report on what the task is waiting on will be displayed:
-
tokio::time::Sleep: the registered timer deadline -
JoinHandle<T>: the id of the awaited task (a dependency edge between tasks) -
tokio::sync::batch_semaphore::Acquire: the contendedMutex/RwLock/Semaphore, their permit counts and wake queue, and associated tasks
An overall view of task relationships is available with graph.
hansei> graph
TASK STATE WAITING ON
619 idle task 765
└─ 765 running future Option<async_bb8_diesel::async_traits::AsyncConnection::run_with_connection::{async_block#0}::{closure_env#0}<async_bb8_diesel::connection::Connection<diesel_dtrace::DTraceConn…
620 idle future core::future::poll_fn::PollFn<oxide_update_engine::engine::StepExec::execute::{async_fn#0}::{closure_env#1}<nexus_types::deployment::execution::spec::ReconfiguratorExecutionSpe…
└─ 2887 [its handle held above] idle future *const alloc::sync::ArcInner<tokio::sync::mpsc::chan::Chan<oxide_update_engine_types::events::Event<nexus_types::deployment::execution::spec::ReconfiguratorExecutionSpec>, toki…
The connections command gives a list of in-flight HTTP1 requests that use the hyper library.
The same --group and --with options can be used here.
hansei> conns ADDR TASK CALLER PROTO ROLE PHASE DEADLINE PEER SERVER METHOD REQUEST 0x11216010 1130 621 http1 client awaiting response — [fd00:1122:3344:10b::2]:12225 — GET http://[fd00:1122:3344:10b::2]:12225/sp/sled/29 0xff8c3d0 1135 — http1 client idle — [fd00:1122:3344:10b::2]:4676 — — — 0xff03710 1143 — http1 client idle — [fd00:1122:3344:108::2]:4676 — — — 0xff72610 1162 — http1 client idle — [fd00:1122:3344:10b::2]:12225 — — — 0x101bc910 1203 — http1 client idle — [fd00:1122:3344:10b::2]:12225 — — — 0xfb719d0 1207 — http1 client idle — [fd00:1122:3344:10b::2]:12225 — — — 0x117f6050 1223 1200 http1 client awaiting response — [fd00:1122:3344:10b::1]:12345 — GET http://[fd00:1122:3344:10b::1]:12345/vmms/9e6bf7ba-a4c7-421c-9481-ff148ebda1d5/state 0x1015ec90 1227 1186 http1 client awaiting response — [fd00:1122:3344:103::1]:12345 — GET http://[fd00:1122:3344:103::1]:12345/vmms/c4f4552f-a0a6-4136-b195-3eafbf022fce/state 0x1223aa10 1535 1498 http1 client awaiting response — [fd00:1122:3344:108::1]:12345 — GET http://[fd00:1122:3344:108::1]:12345/vmms/70d9bfc0-d088-4f30-b7d6-564bbbe52493/state 0x1186da10 1541 1491 http1 client awaiting response — [fd00:1122:3344:101::1]:12345 — GET http://[fd00:1122:3344:101::1]:12345/vmms/fd87e0a0-5e6b-4179-a4ee-e810bd4e98de/state
Use the connection command to view the details of a single connection:
hansei> conn 0x11216010
connection 0x11216010
proto: http1
driven by: task 1130
http: client, awaiting response
dispatcher: 0x11216010 hyper::proto::h1::dispatch::Dispatcher<hyper::proto::h1::dispatch::Client<reqwest::async_impl::body::Body>, reqwest::async_impl::body::Body, reqwest::connect::sealed::Conn, hyper::proto::h1::role::Client>
conn: 0x11216010
keep-alive: busy
reading: init
writing: keep-alive
method: GET
caller: task 621
request: GET http://[fd00:1122:3344:10b::2]:12225/sp/sled/29
alpn: none
proxied: no
tcp: fd 52
stream: 0x111d9218 tokio::net::tcp::stream::TcpStream
registration: 0xff7a780
ready: writable
waker: the reader slot, task 1130
local address: [fd00:1122:3344:127::a3]:46148
peer: [fd00:1122:3344:10b::2]:12225
route:
0x11216088 reqwest::connect::sealed::Conn
0x111d9210 hyper_rustls::stream::MaybeHttpsStream<hyper_util::rt::tokio::TokioIo<tokio::net::tcp::stream::TcpStream>>
0x111d9218 hyper_util::rt::tokio::TokioIo<tokio::net::tcp::stream::TcpStream>
0x111d9218 tokio::net::tcp::stream::TcpStream
---
== Trying it out
`example/` is a small `tokio` program to explore with `hansei`. It runs a number of tasks with
some common features of async Rust programs, including a producer feeding a bounded channel, a
dispatcher, a pool of workers under a `JoinSet`, a fan-out over a `FuturesUnordered`, and a
heartbeat task. It intentionally deadlocks itself and will never exit.
In one terminal, run:
[source,console]
$ cd example && cargo build --release $ ./target/release/hansei-example hansei-example running as pid 1234 core it with: gcore 1234
Then, in another terminal, core the process. On Linux, the binary that executed is used for both `--debug-info` and `--binary`: [source,console]
$ gcore 1234 $ hansei --core core.1234 --binary example/target/release/hansei-example \ --debug-info example/target/release/hansei-example
On illumos the core carries its own symbols, so `--binary` is not needed: [source,console]
$ hansei --core core.1234 --debug-info example/target/release/hansei-example
List the tasks, then filter them by future name with partial matches:
hansei> tasks ID STATE AWAITING AT WAITING ON FUTURE 3 idle src/main.rs:110 io fd 6 (readable) async fn hansei_example::listener 4 idle src/main.rs:139 the semaphore at 0x55b3521b16c0: 1 permit requested, 0 available; wake queue: task 4 async fn hansei_example::producer 5 idle src/main.rs:158 a tokio::sync::Mutex (semaphore 0x55b3521b0cf0): 1 permit requested, 0 available; wake queue: task 5 async fn hansei_example::dispatcher 6 idle src/main.rs:183 future tokio::sync::oneshot::Receiver<()> async fn hansei_example::reloader 7 idle src/main.rs:231 future futures_util::stream::futures_unordered::FuturesUnordered<hansei_example::probe> async fn hansei_example::fanout 8 idle src/main.rs:250 timer (deadline 1661342.991s on the target’s monotonic clock) async fn hansei_example::heartbeat 9 idle src/main.rs:199 future tokio::util::idle_notified_set::IdleNotifiedSet<tokio::runtime::task::join::JoinHandle<usize>> async fn hansei_example::supervisor 10 idle src/main.rs:209 future tokio::sync::notify::Notified async fn hansei_example::worker 11 idle src/main.rs:209 future tokio::sync::notify::Notified async fn hansei_example::worker 12 idle src/main.rs:209 future tokio::sync::notify::Notified async fn hansei_example::worker
hansei> tasks --with type dispatcher ID STATE AWAITING AT WAITING ON FUTURE 5 idle src/main.rs:158 a tokio::sync::Mutex (semaphore 0x55b3521b0cf0): 1 permit requested, 0 available; wake queue: task 5 async fn hansei_example::dispatcher
hansei> tasks --with type worker ID STATE AWAITING AT WAITING ON FUTURE 10 idle src/main.rs:209 future tokio::sync::notify::Notified async fn hansei_example::worker 11 idle src/main.rs:209 future tokio::sync::notify::Notified async fn hansei_example::worker 12 idle src/main.rs:209 future tokio::sync::notify::Notified async fn hansei_example::worker
Here we print the dispatcher's async trace and view the details of one of its locals. Local `config` is an `Arc<Mutex<Config>>`. We walk the `Arc`'s `data` member, then the `Mutex`'s `c` to access the inner config:
hansei> tasks --with type dispatcher --exec trace task 5 #0 future tokio::sync::batch_semaphore::Acquire waiting on a tokio::sync::Mutex (semaphore 0x55b3521b0cf0): 1 permit requested, 0 available; wake queue: task 5 #1 async fn tokio::sync::mutex::Mutex::acquire<hansei_example::Config> awaiting at tokio-1.53.1/src/sync/mutex.rs:658 (Suspend1, 0 locals) #2 async block tokio::sync::mutex::Mutex::lock::{async_fn#0}<hansei_example::Config> awaiting at tokio-1.53.1/src/sync/mutex.rs:436 (Suspend0, 0 locals) #3 async fn tokio::sync::mutex::Mutex::lock<hansei_example::Config> awaiting at tokio-1.53.1/src/sync/mutex.rs:455 (Suspend0, 0 locals) #4 async fn hansei_example::dispatcher awaiting at src/main.rs:158 (Suspend1, 7 locals)
hansei> tasks --with type dispatcher --exec frame 4 print config.data.c task 5 #4 async fn hansei_example::dispatcher awaiting at src/main.rs:158 (Suspend1, 7 locals) hansei_example::Config { upstream: core::net::socket_addr::SocketAddr::V4 { ip: 10.0.0.7, port: 8080, }, max_in_flight: 8, retries: alloc::collections::btree::map::BTreeMap<alloc::string::String, u32, alloc::alloc::Global> { "connect": 3, "read": 1, "write": 2, }, }
`census`, `graph`, and `futures` may be interesting to view as well. == Development On illumos, we build test programs that link against `libproc` to sanity check our cross-platform illumos core dump parsing. `libproc-sys` generates its bindings with `bindgen`, which in turn uses `libclang`. Ensure that the following are set, adjusting the LLVM version as needed: [source,shell]
export LIBCLANG_PATH=/opt/ooce/llvm-15/lib export LD_LIBRARY_PATH=/opt/ooce/llvm-15/lib:$LD_LIBRARY_PATH
The test suite will execute programs and core them, so the user running the tests must be permitted to do this. On Linux this is controlled with `kernel.yama.ptrace_scope`.