aboutsummaryrefslogtreecommitdiff
path: root/kern/slab.c
Commit message (Collapse)AuthorAgeFilesLines
* vm_page_grab: allow allocating in high memorySamuel Thibault2021-08-271-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | vm_page_grab was systematically using the VM_PAGE_SEL_DIRECTMAP selector to play safe with existing code. This adds a flags parameter to let callers of vm_page_grab specify their constraints. Linux drivers need 32bit dmas, Xen drivers use kvtophys to clear some data. Callers of kmem_pagealloc_physmem and vm_page_grab_phys_addr also use kvtophys. Otherwise allocations can go to highmem. This fixes the allocation jam in the directmap segment. * vm/vm_page.h (VM_PAGE_DMA, VM_PAGE_DMA32, VM_PAGE_DIRECTMAP, VM_PAGE_HIGHMEM): New macros. (vm_page_grab): Add flags parameter. * vm/vm_resident.c (vm_page_grab): Choose allocation selector according to flags parameter. (vm_page_convert, vm_page_alloc): Pass VM_PAGE_HIGHMEM to vm_page_grab. (vm_page_grab_phys_addr): Pass VM_PAGE_DIRECTMAP to vm_page_grab. * vm/vm_fault.c (vm_fault_page): Pass VM_PAGE_HIGHMEM to vm_page_grab. * vm/vm_map.c (vm_map_copy_steal_pages): Pass VM_PAGE_HIGHMEM to vm_page_grab. * kern/slab.c (kmem_pagealloc_physmem): Pass VM_PAGE_DIRECTMAP to vm_page_grab. * i386/intel/pmap.c (pmap_page_table_page_alloc): Pass VM_PAGE_DIRECTMAP to vm_page_grab. * xen/block.c (device_read): Pass VM_PAGE_DIRECTMAP to vm_page_grab. * linux/dev/glue/block.c (alloc_buffer): Pass VM_PAGE_DMA32 to vm_page_grab.
* Rework pageout to handle multiple segmentsRichard Braun2016-09-211-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As we're about to use a new HIGHMEM segment, potentially much larger than the existing DMA and DIRECTMAP ones, it's now compulsory to make the pageout daemon aware of those segments. And while we're at it, let's fix some of the defects that have been plaguing pageout forever, such as throttling, and pageout of internal versus external pages (this commit notably introduces a hardcoded policy in which as many external pages are selected before considering internal pages). * kern/slab.c (kmem_pagefree_physmem): Update call to vm_page_release. * vm/vm_page.c: Include <kern/counters.h> and <vm/vm_pageout.h>. (VM_PAGE_SEG_THRESHOLD_MIN_NUM, VM_PAGE_SEG_THRESHOLD_MIN_DENOM, VM_PAGE_SEG_THRESHOLD_MIN, VM_PAGE_SEG_THRESHOLD_LOW_NUM, VM_PAGE_SEG_THRESHOLD_LOW_DENOM, VM_PAGE_SEG_THRESHOLD_LOW, VM_PAGE_SEG_THRESHOLD_HIGH_NUM, VM_PAGE_SEG_THRESHOLD_HIGH_DENOM, VM_PAGE_SEG_THRESHOLD_HIGH, VM_PAGE_SEG_MIN_PAGES, VM_PAGE_HIGH_ACTIVE_PAGE_NUM, VM_PAGE_HIGH_ACTIVE_PAGE_DENOM): New macros. (struct vm_page_queue): New type. (struct vm_page_seg): Add new members `min_free_pages', `low_free_pages', `high_free_pages', `active_pages', `nr_active_pages', `high_active_pages', `inactive_pages', `nr_inactive_pages'. (vm_page_alloc_paused): New variable. (vm_page_pageable, vm_page_can_move, vm_page_remove_mappings): New functions. (vm_page_seg_alloc_from_buddy): Pause allocations and start the pageout daemon as appropriate. (vm_page_queue_init, vm_page_queue_push, vm_page_queue_remove, vm_page_queue_first, vm_page_seg_get, vm_page_seg_index, vm_page_seg_compute_pageout_thresholds): New functions. (vm_page_seg_init): Initialize the new segment members. (vm_page_seg_add_active_page, vm_page_seg_remove_active_page, vm_page_seg_add_inactive_page, vm_page_seg_remove_inactive_page, vm_page_seg_pull_active_page, vm_page_seg_pull_inactive_page, vm_page_seg_pull_cache_page): New functions. (vm_page_seg_min_page_available, vm_page_seg_page_available, vm_page_seg_usable, vm_page_seg_double_lock, vm_page_seg_double_unlock, vm_page_seg_balance_page, vm_page_seg_balance, vm_page_seg_evict, vm_page_seg_compute_high_active_page, vm_page_seg_refill_inactive, vm_page_lookup_seg, vm_page_check): New functions. (vm_page_alloc_pa): Handle allocation failure from VM privileged thread. (vm_page_info_all): Display additional segment properties. (vm_page_wire, vm_page_unwire, vm_page_deactivate, vm_page_activate, vm_page_wait): Move from vm/vm_resident.c and rewrite to use segments. (vm_page_queues_remove, vm_page_check_usable, vm_page_may_balance, vm_page_balance_once, vm_page_balance, vm_page_evict_once): New functions. (VM_PAGE_MAX_LAUNDRY, VM_PAGE_MAX_EVICTIONS): New macros. (vm_page_evict, vm_page_refill_inactive): New functions. * vm/vm_page.h: Include <kern/list.h>. (struct vm_page): Remove member `pageq', reuse the `node' member instead, move the `listq' and `next' members above `vm_page_header'. (VM_PAGE_CHECK): Define as an alias to vm_page_check. (vm_page_check): New function declaration. (vm_page_queue_fictitious, vm_page_queue_active, vm_page_queue_inactive, vm_page_free_target, vm_page_free_min, vm_page_inactive_target, vm_page_free_reserved, vm_page_free_wanted): Remove extern declarations. (vm_page_external_pagedout): New extern declaration. (vm_page_release): Update declaration. (VM_PAGE_QUEUES_REMOVE): Define as an alias to vm_page_queues_remove. (VM_PT_PMAP, VM_PT_KMEM, VM_PT_STACK): Remove macros. (VM_PT_KERNEL): Update value. (vm_page_queues_remove, vm_page_balance, vm_page_evict, vm_page_refill_inactive): New function declarations. * vm/vm_pageout.c (VM_PAGEOUT_BURST_MAX, VM_PAGEOUT_BURST_MIN, VM_PAGEOUT_BURST_WAIT, VM_PAGEOUT_EMPTY_WAIT, VM_PAGEOUT_PAUSE_MAX, VM_PAGE_INACTIVE_TARGET, VM_PAGE_FREE_TARGET, VM_PAGE_FREE_MIN, VM_PAGE_FREE_RESERVED, VM_PAGEOUT_RESERVED_INTERNAL, VM_PAGEOUT_RESERVED_REALLY): Remove macros. (vm_pageout_reserved_internal, vm_pageout_reserved_really, vm_pageout_burst_max, vm_pageout_burst_min, vm_pageout_burst_wait, vm_pageout_empty_wait, vm_pageout_pause_count, vm_pageout_pause_max, vm_pageout_active, vm_pageout_inactive, vm_pageout_inactive_nolock, vm_pageout_inactive_busy, vm_pageout_inactive_absent, vm_pageout_inactive_used, vm_pageout_inactive_clean, vm_pageout_inactive_dirty, vm_pageout_inactive_double, vm_pageout_inactive_cleaned_external): Remove variables. (vm_pageout_requested, vm_pageout_continue): New variables. (vm_pageout_setup): Wait for page allocation to succeed instead of falling back to flush, update double paging protocol with caller, add pageout throttling setup. (vm_pageout_scan): Rewrite to use the new vm_page balancing, eviction and inactive queue refill functions. (vm_pageout_scan_continue, vm_pageout_continue): Remove functions. (vm_pageout): Rewrite. (vm_pageout_start, vm_pageout_resume): New functions. * vm/vm_pageout.h (vm_pageout_continue, vm_pageout_scan_continue): Remove function declarations. (vm_pageout_start, vm_pageout_resume): New function declarations. * vm/vm_resident.c: Include <kern/list.h>. (vm_page_queue_fictitious): Define as a struct list. (vm_page_free_wanted, vm_page_external_count, vm_page_free_avail, vm_page_queue_active, vm_page_queue_inactive, vm_page_free_target, vm_page_free_min, vm_page_inactive_target, vm_page_free_reserved): Remove variables. (vm_page_external_pagedout): New variable. (vm_page_bootstrap): Don't initialize removed variable, update initialization of vm_page_queue_fictitious. (vm_page_replace): Call VM_PAGE_QUEUES_REMOVE where appropriate. (vm_page_remove): Likewise. (vm_page_grab_fictitious): Update to use list_xxx functions. (vm_page_release_fictitious): Likewise. (vm_page_grab): Remove pageout related code. (vm_page_release): Add `laundry' and `external' parameters for pageout throttling. (vm_page_grab_contig): Remove pageout related code. (vm_page_free_contig): Likewise. (vm_page_free): Remove pageout related code, update call to vm_page_release. (vm_page_wait, vm_page_wire, vm_page_unwire, vm_page_deactivate, vm_page_activate): Move to vm/vm_page.c.
* Redefine what an external page isRichard Braun2016-09-211-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of a "page considered external", which apparently takes into account whether a page is dirty or not, redefine this property to reliably mean "is in an external object". This commit mostly deals with the impact of this change on the page allocation interface. * i386/intel/pmap.c (pmap_page_table_page_alloc): Update call to vm_page_grab. * kern/slab.c (kmem_pagealloc_physmem): Use vm_page_grab instead of vm_page_grab_contig. (kmem_pagefree_physmem): Use vm_page_release instead of vm_page_free_contig. * linux/dev/glue/block.c (alloc_buffer, device_read): Update call to vm_page_grab. * vm/vm_fault.c (vm_fault_page): Update calls to vm_page_grab and vm_page_convert. * vm/vm_map.c (vm_map_copy_steal_pages): Update call to vm_page_grab. * vm/vm_page.h (struct vm_page): Remove `extcounted' member. (vm_page_external_limit, vm_page_external_count): Remove extern declarations. (vm_page_convert, vm_page_grab): Update declarations. (vm_page_release, vm_page_grab_phys_addr): New function declarations. * vm/vm_pageout.c (VM_PAGE_EXTERNAL_LIMIT): Remove macro. (VM_PAGE_EXTERNAL_TARGET): Likewise. (vm_page_external_target): Remove variable. (vm_pageout_scan): Remove specific handling of external pages. (vm_pageout): Don't set vm_page_external_limit and vm_page_external_target. * vm/vm_resident.c (vm_page_external_limit): Remove variable. (vm_page_insert, vm_page_replace, vm_page_remove): Update external page tracking. (vm_page_convert): Remove `external' parameter. (vm_page_grab): Likewise. Remove specific handling of external pages. (vm_page_grab_phys_addr): Update call to vm_page_grab. (vm_page_release): Remove `external' parameter and remove specific handling of external pages. (vm_page_wait): Remove specific handling of external pages. (vm_page_alloc): Update call to vm_page_grab. (vm_page_free): Update call to vm_page_release. * xen/block.c (device_read): Update call to vm_page_grab. * xen/net.c (device_write): Likewise.
* Fix page fault in critical section in the slab allocatorRichard Braun2016-06-291-28/+36
| | | | | * kern/slab.c (host_slab_info): Use wired kernel memory to build the cache info.
* Fix locking error in the slab allocatorRichard Braun2016-06-291-19/+18
| | | | | | | * kern/slab.c (kmem_slab_create): Set `slab->cache` member. (kmem_cache_reap): Return dead slabs instead of destroying in place. (slab_collect): Destroy slabs outside of critical section. * kern/slab.h (struct kmem_slab): New `cache` member.
* Relax slab allocation alignment constraintRichard Braun2016-03-091-8/+12
| | | | | | | | | | | * kern/slab.c (kmem_pagealloc_virtual): Pass alignment to function, call kmem_alloc_aligned when greater than a page. (kmem_pagealloc): Pass alignment to function. (kmem_slab_create): Update call to kmem_pagealloc. (kalloc): Likewise. (kmem_cache_compute_properties): Fix handling of color with large slab sizes. (kmem_cache_init): Allow alignment greater than the page size.
* Remove kmem cache flags from the debugging interfaceRichard Braun2016-02-221-6/+1
| | | | | | | * include/mach_debug/slab_info.h (CACHE_FLAGS_NO_CPU_POOL, CACHE_FLAGS_SLAB_EXTERNAL, CACHE_FLAGS_NO_RECLAIM, CACHE_FLAGS_VERIFY, CACHE_FLAGS_DIRECT): Remove macros. * kern/slab.c (host_slab_info): Pass raw cache flags to caller.
* Fix slab allocator option handlingRichard Braun2016-02-221-49/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The slab allocator has grown to use multiple ways to allocate slabs as well as track them, which got a little messy. One consequence is the breaking of the KMEM_CF_VERIFY option. In order to make the code less confusing, this change expresses all options as explicit cache flags and clearly defines their relationships. The special kmem_slab and vm_map_entry caches are initialized accordingly. * kern/slab.c (KMEM_CF_DIRECTMAP): Rename to ... (KMEM_CF_PHYSMEM): ... this new macro. (KMEM_CF_DIRECT): Restore macro. (KMEM_CF_USE_TREE, KMEM_CF_USE_PAGE): New macros. (KMEM_CF_VERIFY): Update value. (kmem_pagealloc_directmap): Rename to... (kmem_pagealloc_physmem): ... this new function. (kmem_pagefree_directmap): Rename to ... (kmem_pagefree_physmem): ... this new function. (kmem_pagealloc, kmem_pagefree): Update macro names. (kmem_slab_use_tree): Remove function. (kmem_slab_create, kmem_slab_destroy): Update according to the new cache flags. (kmem_cache_compute_sizes): Rename to ... (kmem_cache_compute_properties): ... this new function, and update to properly set cache flags. (kmem_cache_init): Update call to kmem_cache_compute_properties. (kmem_cache_alloc_from_slab): Check KMEM_CF_USE_TREE instead of calling the defunct kmem_slab_use_tree function. (kmem_cache_free_to_slab): Update according to the new cache flags. kmem_cache_free_verify): Add assertion. (slab_init): Update initialization of kmem_slab_cache. * kern/slab.h (KMEM_CACHE_DIRECTMAP): Rename to ... (KMEM_CACHE_PHYSMEM): ... this new macro. * vm/vm_map.c (vm_map_init): Update initialization of vm_map_entry_cache.
* Optimize slab lookup on the free pathRichard Braun2016-02-221-11/+43
| | | | | | | | | | | | | Caches that use external slab data but allocate slabs from the direct physical mapping can look up slab data in constant time by associating the slab data directly with the underlying page. * kern/slab.c (kmem_slab_use_tree): Take KMEM_CF_DIRECTMAP into account. (kmem_slab_create): Set page private data if relevant. (kmem_slab_destroy): Clear page private data if relevant. (kmem_cache_free_to_slab): Use page private data if relevant. * vm/vm_page.c (vm_page_init_pa): Set `priv' member to NULL. * vm/vm_page.h (vm_page_set_priv, vm_page_get_priv): New functions.
* Fix unused variable warningsRichard Braun2016-02-221-2/+0
| | | | * kern/slab.c (slab_init): Remove unused variables.
* Avoid slab allocation failures caused by memory fragmentationRichard Braun2016-02-201-62/+91
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since the slab allocator has been changed to sit directly on top of the physical allocator, failures caused by fragmentation have been observed, as one could expect. This change makes the slab allocator revert to kernel virtual memory when allocating larger-than-page slabs. This solution is motivated in part to avoid the complexity of other solutions such as page mobility, and also because a microkernel cannot be extended to new arbitrary uncontrolled usage patterns such as a monolithic kernel with loadable modules. As such, large objects are rare, and their use infrequent, which is compatible with the use of kernel virtual memory. * kern/slab.c: Update module description. (KMEM_CF_SLAB_EXTERNAL, KMEM_CF_VERIFY): Update values. (KMEM_CF_DIRECT): Remove macro. (KMEM_CF_DIRECTMAP): New macro. (kmem_pagealloc_directmap, kmem_pagefree_directmap, kmem_pagealloc_virtual, kmem_pagefree_virtual): New functions. (kmem_pagealloc, kmem_pagefree, kmem_slab_create, kmem_slab_destroy, kalloc, kfree): Update to use the new pagealloc functions. (kmem_cache_compute_sizes): Update the algorithm used to determine slab size and other cache properties. (kmem_slab_use_tree, kmem_cache_free_to_slab, host_slab_info): Update to correctly use the cache flags. (slab_init): Add KMEM_CACHE_DIRECTMAP to the kmem_slab_cache init flags. * kern/slab.h (KMEM_CACHE_VERIFY): Change value. (KMEM_CACHE_DIRECTMAP): New macro. * vm/vm_map.c (vm_map_init): Add KMEM_CACHE_DIRECTMAP to the vm_map_entry_cache init flags.
* Remove kmem mapRichard Braun2016-02-071-13/+0
| | | | | | | | | | Now that the slab allocator doesn't use kernel virtual memory any more, this map has become irrelevant. * kern/slab.c (KMEM_MAP_SIZE): Remove macro. (kmem_map_store, kmem_map): Remove variables. (slab_init): Remove call kmem_submap. * kern/slab.h (kmem_map): Remove extern declaration.
* Change computation of slab sizeRichard Braun2016-02-061-56/+26
| | | | | | | | | | | | Allocating directly out of the physical memory allocator makes the slab allocator vulnerable to failures due to fragmentation. This change makes the slab allocator use the lowest possible size for its slabs to reduce the chance of contiguous allocation failures. * kern/slab.c (KMEM_MIN_BUFS_PER_SLAB, KMEM_SLAB_SIZE_THRESHOLD): Remove macros. (kmem_cache_compute_sizes): Update the algorithm used to determine slab size and other cache properties.
* Fix various memory managment errorsRichard Braun2016-02-021-12/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A few errors were introduced in the latest changes. o Add VM_PAGE_WAIT calls around physical allocation attempts in case of memory exhaustion. o Fix stack release. o Fix memory exhaustion report. o Fix free page accounting. * kern/slab.c (kmem_pagealloc, kmem_pagefree): New functions (kmem_slab_create, kmem_slab_destroy, kalloc, kfree): Use kmem_pagealloc and kmem_pagefree instead of the raw page allocation functions. (kmem_cache_compute_sizes): Don't store slab order. * kern/slab.h (struct kmem_cache): Remove `slab_order' member. * kern/thread.c (stack_alloc): Call VM_PAGE_WAIT in case of memory exhaustion. (stack_collect): Call vm_page_free_contig instead of kmem_free to release pages. * vm/vm_page.c (vm_page_seg_alloc): Fix memory exhaustion report. (vm_page_setup): Don't update vm_page_free_count. (vm_page_free_pa): Check page parameter. (vm_page_mem_free): New function. * vm/vm_page.h (vm_page_free_count): Remove extern declaration. (vm_page_mem_free): New prototype. * vm/vm_pageout.c: Update comments not to refer to vm_page_free_count. (vm_pageout_scan, vm_pageout_continue, vm_pageout): Use vm_page_mem_free instead of vm_page_free_count, update types accordingly. * vm/vm_resident.c (vm_page_free_count, vm_page_free_count_minimum): Remove variables. (vm_page_free_avail): New variable. (vm_page_bootstrap, vm_page_grab, vm_page_release, vm_page_grab_contig, vm_page_free_contig, vm_page_wait): Use vm_page_mem_free instead of vm_page_free_count, update types accordingly, don't set vm_page_free_count_minimum. * vm/vm_user.c (vm_statistics): Likewise.
* Stack the slab allocator directly on top of the physical allocatorRichard Braun2016-02-021-93/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In order to increase the amount of memory available for kernel objects, without reducing the amount of memory available for user processes, a new allocation strategy is introduced in this change. Instead of allocating kernel objects out of kernel virtual memory, the slab allocator directly uses the direct mapping of physical memory as its backend. This largely increases the kernel heap, and removes the need for address translation updates. In order to allow this strategy, an assumption made by the interrupt code had to be removed. In addition, kernel stacks are now also allocated directly from the physical allocator. * i386/i386/db_trace.c: Include i386at/model_dep.h (db_i386_reg_value): Update stack check. * i386/i386/locore.S (trap_from_kernel, all_intrs, int_from_intstack): Update interrupt handling. * i386/i386at/model_dep.c: Include kern/macros.h. (int_stack, int_stack_base): New variables. (int_stack_high): Remove variable. (i386at_init): Update interrupt stack initialization. * i386/i386at/model_dep.h: Include i386/vm_param.h. (int_stack_top, int_stack_base): New extern declarations. (ON_INT_STACK): New macro. * kern/slab.c: Include vm/vm_page.h (KMEM_CF_NO_CPU_POOL, KMEM_CF_NO_RECLAIM): Remove macros. (kmem_pagealloc, kmem_pagefree, kalloc_pagealloc, kalloc_pagefree): Remove functions. (kmem_slab_create): Allocate slab pages directly from the physical allocator. (kmem_slab_destroy): Release slab pages directly to the physical allocator. (kmem_cache_compute_sizes): Update the slab size computation algorithm to return a power-of-two suitable for the physical allocator. (kmem_cache_init): Remove custom allocation function pointers. (kmem_cache_reap): Remove check on KMEM_CF_NO_RECLAIM. (slab_init, kalloc_init): Update calls to kmem_cache_init. (kalloc, kfree): Directly fall back on the physical allocator for big allocation sizes. (host_slab_info): Remove checks on defunct flags. * kern/slab.h (kmem_slab_alloc_fn_t, kmem_slab_free_fn_t): Remove types. (struct kmem_cache): Add `slab_order' member, remove `slab_alloc_fn' and `slab_free_fn' members. (KMEM_CACHE_NOCPUPOOL, KMEM_CACHE_NORECLAIM): Remove macros. (kmem_cache_init): Update prototype, remove custom allocation functions. * kern/thread.c (stack_alloc): Allocate stacks from the physical allocator. * vm/vm_map.c (vm_map_kentry_cache, kentry_data, kentry_data_size): Remove variables. (kentry_pagealloc): Remove function. (vm_map_init): Update calls to kmem_cache_init, remove initialization of vm_map_kentry_cache. (vm_map_create, _vm_map_entry_dispose, vm_map_copyout): Unconditionnally use vm_map_entry_cache. * vm/vm_map.h (kentry_data, kentry_data_size, kentry_count): Remove extern declarations. * vm/vm_page.h (VM_PT_STACK): New page type. * device/dev_lookup.c (dev_lookup_init): Update calls to kmem_cache_init. * device/dev_pager.c (dev_pager_hash_init, device_pager_init): Likewise. * device/ds_routines.c (mach_device_init, mach_device_trap_init): Likewise. * device/net_io.c (net_io_init): Likewise. * i386/i386/fpu.c (fpu_module_init): Likewise. * i386/i386/machine_task.c (machine_task_module_init): Likewise. * i386/i386/pcb.c (pcb_module_init): Likewise. * i386/intel/pmap.c (pmap_init): Likewise. * ipc/ipc_init.c (ipc_bootstrap): Likewise. * ipc/ipc_marequest.c (ipc_marequest_init): Likewise. * kern/act.c (global_act_init): Likewise. * kern/processor.c (pset_sys_init): Likewise. * kern/rdxtree.c (rdxtree_cache_init): Likewise. * kern/task.c (task_init): Likewise. * vm/memory_object_proxy.c (memory_object_proxy_init): Likewise. * vm/vm_external.c (vm_external_module_initialize): Likewise. * vm/vm_fault.c (vm_fault_init): Likewise. * vm/vm_object.c (vm_object_bootstrap): Likewise. * vm/vm_resident.c (vm_page_module_init): Likewise. (vm_page_bootstrap): Remove initialization of kentry_data.
* Increase kernel map sizeSamuel Thibault2016-01-131-1/+1
| | | | | | | | To avoid running out of memory due to the increased consumption by radix trees which replaced arrays. * kern/slab.c (KMEM_MAP_SIZE): Bump from 96MiB to 128MiB. * i386/i386/vm_param.h (VM_KERNEL_MAP_SIZE): Add 32MiB accordingly.
* kern/slab: print total used and reclaimable memoryJustus Winter2015-09-291-1/+10
| | | | * kern/slab.c (_slab_info): Print total used and reclaimable memory.
* kern/slab: include flags in the slab informationJustus Winter2015-09-291-6/+7
| | | | * kern/slab.c (_slab_info): Include flags in the slab information.
* ddb: add new command `show slabinfo'Justus Winter2015-09-291-3/+18
| | | | | | | | | * ddb/db_command.c (db_show_cmds): Add `slabinfo'. * kern/slab.c (slab_info): Generalize so that it can be used with different printf-like functions, and turn it into a static function. (slab_info): New wrapper retaining the old behaviour. (db_show_slab_info): New wrapper that uses `db_printf' instead. * kern/slab.h (db_show_slab_info): New declaration.
* kern/slab: fix lockingJustus Winter2015-07-181-1/+1
| | | | * kern/slab.c (host_slab_info): Fix locking.
* kern: import `macros.h' from x15Justus Winter2015-05-191-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Import the macro definitions from the x15 kernel project, and replace all similar definitions littered all over the place with it. Importing this file will make importing code from the x15 kernel easier. We are already using the red-black tree implementation and the slab allocator from it, and we will import even more code in the near future. * kern/list.h: Do not define `structof', include `macros.h' instead. * kern/rbtree.h: Likewise. * kern/slab.c: Do not define `ARRAY_SIZE', include `macros.h' instead. * i386/grub/misc.h: Likewise. * i386/i386/xen.h: Do not define `barrier', include `macros.h' instead. * kern/macro_help.h: Delete file. Replaced by `macros.h'. * kern/macros.h: New file. * Makefrag.am (libkernel_a_SOURCES): Add new file, remove old file. * device/dev_master.h: Adopt accordingly. * device/io_req.h: Likewise. * device/net_io.h: Likewise. * i386/intel/read_fault.c: Likewise. * ipc/ipc_kmsg.h: Likewise. * ipc/ipc_mqueue.h: Likewise. * ipc/ipc_object.h: Likewise. * ipc/ipc_port.h: Likewise. * ipc/ipc_space.h: Likewise. * ipc/ipc_splay.c: Likewise. * ipc/ipc_splay.h: Likewise. * kern/assert.h: Likewise. * kern/ast.h: Likewise. * kern/pc_sample.h: Likewise. * kern/refcount.h: Likewise. * kern/sched.h: Likewise. * kern/sched_prim.c: Likewise. * kern/timer.c: Likewise. * kern/timer.h: Likewise. * vm/vm_fault.c: Likewise. * vm/vm_map.h: Likewise. * vm/vm_object.h: Likewise. * vm/vm_page.h: Likewise.
* kern: silence compiler warning about uninitialized variableJustus Winter2014-09-301-1/+2
| | | | | * kern/slab.c (kmem_cache_compute_sizes): Initialize optimal_size and assert that a size is selected.
* Reduce kmem_map to make room for kentry_data_sizeSamuel Thibault2014-02-121-1/+1
| | | | * kern/slab.c (KMEM_MAP_SIZE): Decrease from 128MiB to 96MiB.
* kern: make kmem_error panicJustus Winter2014-02-041-2/+2
| | | | | | | | | The slab allocator relies on the fact that kmem_cache_error does not return. Previously, kmem_error was using printf. Use panic instead. Found using the Clang Static Analyzer. * kern/slab.c (kmem_error): Use panic instead of printf.
* kern: use kmem_warn instead of kmem_error in kmem_cache_errorJustus Winter2014-02-041-1/+1
| | | | | * kern/slab.c (kmem_cache_error): Use kmem_warn instead of kmem_error to print the cache name and its address.
* kern/slab.c: initialize info_sizeMarin Ramesa2013-11-191-1/+1
| | | | | | | info_size is initialized to a random value. Quiet the warning by initializing to zero. * kern/slab.c (info_size): Initialize to zero.
* kern/slab.c: initialize optimal_embedMarin Ramesa2013-11-191-1/+1
| | | | | | | optimal_embed is initialized to a random value. Quiet the warning by initializing to zero. * kern/slab.c (optimal_embed): Initialize to zero.
* (slab_info): fix format warningsMarin Ramesa2013-06-291-1/+1
| | | | * kern/slab.c (slab_info): Fix format for vm_size_t.
* Fix object construction in the slab allocatorRichard Braun2013-06-091-4/+10
| | | | | | | | There is currently no actual use of constructors, which is why this bug has been long overlooked. * kern/slab.c (kmem_cpu_pool_fill): Call constructor on buffers unless verification is enabled for the cache, or the constructor is NULL.
* Fix yet another locking error in the slab allocatorRichard Braun2013-06-021-0/+1
| | | | | * kern/slab.c (kmem_cache_free): Relock cache before retrying releasing an object to the CPU pool layer.
* Reduce fragmentation in the slab allocatorRichard Braun2013-05-161-2/+5
| | | | | | | | | | This reverts a change brought when reworking slab lists handling that made the allocator store slabs in LIFO order, whatever their reference count. While it's fine for free slabs, it actually increased fragmentation for partial slabs. * kern/slab.c (kmem_cache_alloc_from_slab): Insert slabs that become partial at the end of the partial slabs list.
* Rename list_insert to list_insert_headRichard Braun2013-05-161-4/+4
| | | | | | | | This change increases clarity. * kern/list.h (list_insert): Rename to ... (list_insert_head): ... this. * kern/slab.c: Update calls to list_insert.
* Drop unused variablesMiguel Figueiredo2013-05-131-1/+0
| | | | * kern/slab.c (kalloc_init): Remove unused variables.
* Optimize slab reapingRichard Braun2013-04-211-13/+10
| | | | | | | | Instead of walking the list of free slabs while holding the cache lock, detach the list from the cache and directly compute the final count values, and destroy slabs after releasing the cache lock. * kern/slab.c (kmem_cache_reap): Optimize.
* Rework slab lists handlingRichard Braun2013-04-211-74/+8
| | | | | | | | | | | | Don't enforce strong ordering of partial slabs. Separating partial slabs from free slabs is already effective against fragmentation, and sorting would sometimes cause pathological scalability issues. In addition, store new slabs (whether free or partial) in LIFO order for better cache usage. * kern/slab.c (kmem_cache_grow): Insert new slab at the head of the slabs list. (kmem_cache_alloc_from_slab): Likewise. In addition, don't sort partial slabs. (kmem_cache_free_to_slab): Likewise. * kern/slab.h: Remove comment about partial slabs sorting.
* Fix locking error in the slab allocatorRichard Braun2013-04-211-0/+2
| | | | | * kern/slab.c (kmem_cache_free): Lock cache before releasing an object to the slab layer.
* Fix slab cache list lockingRichard Braun2013-01-081-4/+4
| | | | | | | This problem was overlooked because of simple locks being no-ops. * kern/slab.c (slab_collect): Fix lock name to kmem_cache_list_lock. (host_slab_info): Likewise.
* Add function to dump a raw summary of the slab allocator stateRichard Braun2013-01-081-0/+29
| | | | | | | | | The purpose of this function is to allow kernel code to display the state of the slab caches in situations where the host_slab_info RPC wouldn't be available, e.g. before a panic. * kern/slab.c (slab_info): New function. * kern/slab.h: Add declaration for slab_info.
* Fix slab collection timingRichard Braun2012-07-081-4/+4
| | | | | | | | | The slab garbage collector uses sched_tick as its time reference, which is increased every seconds, while the interval is expressed in clock ticks. Use the proper time reference instead. * kern/slab.c (kmem_gc_last_tick): Declare as unsigned long. (slab_collect): Use elapsed_ticks instead of sched_tick.
* Increase the slab collection intervalRichard Braun2012-07-071-1/+1
| | | | * kern/slab.c (KMEM_GC_INTERVAL): Increase to 5 seconds.
* Merge kalloc_map into kmem_mapRichard Braun2012-07-071-17/+4
| | | | | | | | | | | | | | | | | * ipc/ipc_table.c: Add #include <kern/slab.h>. (ipc_table_alloc): Use kmem_map instead of kalloc_map when allocating a table. (ipc_table_realloc): Likewise for reallocation. (ipc_table_free): Likewise for release. * kern/kalloc.h (kalloc_map): Remove declaration. * kern/slab.c (KMEM_MAP_SIZE): Increase to 128 MiB. (KALLOC_MAP_SIZE): Remove macro. (kalloc_map_store): Remove variable. (kalloc_map): Likewise. (kalloc_pagealloc): Use kmem_map instead of kalloc_map for general purpose allocations. (kalloc_pagefree): Likewise. (kalloc_init): Remove the creation of kalloc_map.
* Fix copyright assignmentRichard Braun2012-04-221-2/+25
| | | | | | | | | | | | Maksym and I have assigned copyright to the Free Software Foundation. In addition, restore the original upstream copyrights for correctness. * kern/list.h: Fix copyright assignment. * kern/rbtree.c: Likewise. * kern/rbtree.h: Likewise. * kern/rbtree_i.h: Likewise. * kern/slab.c: Likewise. * kern/slab.h: Likewise.
* Use unsigned long for addresses and sizesSamuel Thibault2012-03-091-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | TODO: remonter formats * i386/include/mach/i386/vm_types.h (vm_offset_t): Define to unsigned long. (signed32_t): Define to signed int. (unsigned32_t): Define to unsigned int. * i386/include/mach/sa/stdarg.h (__va_size): Use sizeof(unsigned long)-1 instead of 3. * include/mach/port.h (mach_port_t): Define to vm_offset_t instead of natural_t. * include/sys/types.h (size_t): Define to unsigned long instead of natural_t. * linux/src/include/asm-i386/posix_types.h (__kernel_size_t): Define to unsigned long. (__kernel_ssize_t): Define to long. * linux/src/include/linux/stddef.h (size_t): Define to unsigned long. * device/dev_pager.c (dev_pager_hash): Cast port to vm_offset_t insted of natural_t. (device_pager_data_request): Fix format. * device/ds_routines.c (ds_no_senders): Fix format. * i386/i386/io_map.c (io_map): Likewise. * i386/i386at/autoconf.c (take_dev_irq): Likewise. * i386/i386at/com.c (comattach): Likewise. * i386/i386at/lpr.c (lprattach): Likewise. * i386/i386at/model_dep.c (mem_size_init, mem_size_init, c_boot_entry): Likewise. * i386/intel/pmap.c (pmap_enter): Likewise. * ipc/ipc_notify.c (ipc_notify_port_deleted, ipc_notify_msg_accepted, ipc_notify_dead_name): Likewise. * ipc/mach_port.c (mach_port_destroy, mach_port_deallocate): Likewise. * kern/ipc_kobject.c (ipc_kobject_destroy): Likewise. * kern/slab.c (kalloc_init): Likewise. * vm/vm_fault.c (vm_fault_page): Likewise. * vm/vm_map.c (vm_map_pmap_enter): Likewise. * xen/block.c (device_read): Likewise. * device/net_io.c (bpf_match): Take unsigned long * instead of unsigned int *. (bpf_do_filter): Make mem unsigned long instead of long. * i386/i386/ktss.c (ktss_init): Cast pointer to unsigned long instead of unsigned. * i386/i386/pcb.c (stack_attach, switch_ktss): Cast pointers to long instead of int. * i386/i386/trap.c (dump_ss): Likewise. * ipc/ipc_hash.c (IH_LOCAL_HASH): Cast object to vm_offset_t. * ipc/mach_msg.c (mach_msg_receive, mach_msg_receive_continue): Cast kmsg to vm_offset_t instead of natural_t. * kern/pc_sample.c (take_pc_sample): Cast to vm_offset_t instead of natural_t. * kern/boot_script.c (sym, arg): Set type of `val' field to long instead of int. (create_task, builtin_symbols, boot_script_parse_line, boot_script_define_function): Cast to long instead of int. * kern/bootstrap.c (bootstrap_create): Likewise. * kern/sched_prim.c (decl_simple_lock_data): Likewise. * kern/printf.c (vsnprintf): Set size type to size_t. * kern/printf.h (vsnprintf): Likewise. * vm/vm_map.h (kentry_data_size): Fix type to vm_size_t. * vm/vm_object.c (vm_object_pmap_protect_by_page): Fix size parameter type to vm_size_t.
* Import the slab allocatorRichard Braun2011-12-171-0/+1576
As it is intended to completely replace the zone allocator, remove it on the way. So long to the venerable code ! * Makefrag.am (libkernel_a_SOURCES): Add kern/slab.{c,h}, remove kern/kalloc.c and kern/zalloc.{c,h}. * configfrag.ac (SLAB_VERIFY, SLAB_USE_CPU_POOLS): Add defines. * i386/Makefrag.am (libkernel_a_SOURCES): Remove i386/i386/zalloc.h. * i386/configfrag.ac (CPU_L1_SHIFT): Remove define. * include/mach_debug/slab_info.h: New file. * kern/slab.c: Likewise. * kern/slab.h: Likewise. * i386/i386/zalloc.h: Remove file. * include/mach_debug/zone_info.h: Likewise. * kern/kalloc.c: Likewise. * kern/zalloc.c: Likewise. * kern/zalloc.h: Likewise.
generated by cgit v1.2.3 (git 2.39.1) at 2025-04-08 03:29:30 +0000