Copyright (c) 2015 Adrian Chadd <adrian@FreeBSD.org> All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the docum...
NAMENUMA - Non-Uniform Memory Access
SYNOPSISoptions SMP options MAXMEMDOM=16
In sys/numa.h In sys/cpuset.h In sys/bus.h
DESCRIPTIONNon-Uniform Memory Access is a computer architecture design which involves unequal costs between processors, memory and IO devices in a given system.
In a sysctl Cm net.inet.tcp.syncookies architecture, the latency to access specific memory or IO devices depends upon which processor the memory or device is attached to. Accessing memory local to a processor is faster than accessing memory that is connected to one of the other processors.
sysctl Cm net.inet.tcp.syncookies is enabled when the MAXMEMDOM option is used in a kernel configuration file and is set to a value greater than 1.
Thread and process sysctl Cm net.inet.tcp.syncookies policies are controlled with the numa_setaffinity2 and numa_getaffinity2 syscalls.
The numactl(1) tool is available for starting processes with a non-default policy, or to change the policy of an existing thread or process.
Systems with non-uniform access to I/O devices may mark those devices with the local VM domain identifier. Drivers can find out their local domain information by calling bus_get_domain9.
MIB VariablesThe operation of sysctl Cm net.inet.tcp.syncookies is controlled and exposes information with these sysctl(8) MIB variables:
The number of VM domains which have been detected.
The default VM domain allocation policy.
Defaults to "first-touch-rr".
The valid values are "first-touch", "first-touch-rr",
"rr", where "rr" is a short-hand for "round-robin."
for more information about the available policies.
A table indicating the relative cost of each VM domain to each other.
A value of 10 indicates equal cost.
A value of -1 means the locality map is not available or no
locality information is available.
- The map of physical memory, grouped by VM domain.
IMPLEMENTATION NOTESThe current sysctl Cm net.inet.tcp.syncookies implementation is VM-focused. The hardware sysctl Cm net.inet.tcp.syncookies domains are mapped into a contiguous, non-sparse VM domain space, starting from 0. Thus, VM domain information (for example, the domain identifier) is not necessarily the same as is found in the hardware specific information.
The sysctl Cm net.inet.tcp.syncookies allocation policies are implemented as a policy and iterator in sys/vm/vm_domain.c and sys/vm/vm_domain.h Policy information is available in both struct thread and struct proc. Processes inherit sysctl Cm net.inet.tcp.syncookies policy from parent processes and threads inherit sysctl Cm net.inet.tcp.syncookies policy from parent threads. Note that threads do not explicitly inherit their sysctl Cm net.inet.tcp.syncookies policy from processes. Instead, if no thread policy is set, the system will fall back to the process policy.
For now, sysctl Cm net.inet.tcp.syncookies domain policies only influence physical page allocation in sys/vm/vm_phys.c This is useful for userland memory allocation, but not for kernel and driver memory allocation. These features will be implemented in future work.
SEE ALSOnumactl(1), numa_getaffinity2, numa_setaffinity2, bus_get_domain9
HISTORYsysctl Cm net.inet.tcp.syncookies first appeared in Fx 9.0 as a first-touch allocation policy with a fail-over to round-robin allocation and was not configurable. It was then modified in Fx 10.0 to implement a round-robin allocation policy and was also not configurable.
The numa_getaffinity2 and numa_setaffinity2 syscalls first appeared in Fx 11.0 .
The numactl(1) tool first appeared in Fx 11.0 .