diff options
Diffstat (limited to 'microkernel')
52 files changed, 2021 insertions, 456 deletions
diff --git a/microkernel/barrelfish.mdwn b/microkernel/barrelfish.mdwn new file mode 100644 index 00000000..8cf5591b --- /dev/null +++ b/microkernel/barrelfish.mdwn @@ -0,0 +1,24 @@ +[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +<http://barrelfish.org/> + + * {{$fof_plos09}} + + +[[!ymlfront data=""" + +fof_plos09: + + "Pierre-Evariste Dagand, Andrew Baumann, Timothy Roscoe. Filet-o-Fish: + practical and dependable domain-specific languages for OS development. PLOS + '09, October 11, 2009, Big Sky, Montana, USA." + +"""]] diff --git a/microkernel/coyotos.mdwn b/microkernel/coyotos.mdwn new file mode 100644 index 00000000..fec023ba --- /dev/null +++ b/microkernel/coyotos.mdwn @@ -0,0 +1,33 @@ +[[!meta copyright="Copyright © 2006, 2007, 2008, 2010, 2011 Free Software +Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Coyotos"]] + +[*Coyotos*](http://www.coyotos.org/) is a microkernel and OS and the successor +of [[EROS]], that itself is the successor of [[KeyKOS]]. A more complete +history can be found [here](http://www.coyotos.org/history.html). Its main +objectives are to correcte some shortcomings of [[EROS]], demonstrate that an +atomic kernel design scales well, and (eventually) to completely formally +verify both the kernel and critical system components by writing them in a new +language called [bitc](http://www.bitc-lang.org/). + +Coyotos is an orthogonally [[persistent|persistency]] pure [[capability]] +system. It uses [[continuation]]-based unbuffered asynchronous [[IPC]] +(actually it's synchronous [[IPC]] with asynchronous [[system calls]]). + +TODO: explain these terms and (more important) their consequences on system +design. + +The coyotos microkernel specification can be found +[here](http://www.coyotos.org/docs/ukernel/spec.html). + +There once was the idea of a GNU/Hurd [[port using the Coyotos +microkernel|history/port_to_another_microkernel]], but this didn't come live. diff --git a/microkernel/discussion.mdwn b/microkernel/discussion.mdwn new file mode 100644 index 00000000..a5a73e18 --- /dev/null +++ b/microkernel/discussion.mdwn @@ -0,0 +1,24 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + +IRC, freenode, #hurd, 2011-07-26: + + < antrik> Tekk_`: regarding microkernels: the basic idea, and really the + *only* fundamental difference, is that they isolate things in separate + address spaces. everything else goes back to this. + < antrik> benefits from the isolation generally fall into two groups: more + robustness (main focus of Minix3), and more flexibility (main focus of + Hurd) + < antrik> while it might also encourage some other good design choices, + these are secondary effects: such choices can also be implemented in a + monolithic architecture -- and not necessarily harder. just less obvious + in some cases... diff --git a/microkernel/eros.mdwn b/microkernel/eros.mdwn new file mode 100644 index 00000000..be1ca90a --- /dev/null +++ b/microkernel/eros.mdwn @@ -0,0 +1,15 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + +<http://www.eros-os.org/> + +TODO. <http://www.eros-os.org/essays/reliability/paper.html> diff --git a/microkernel/faq.mdwn b/microkernel/faq.mdwn index a6c4f1f8..fe259f05 100644 --- a/microkernel/faq.mdwn +++ b/microkernel/faq.mdwn @@ -1,4 +1,5 @@ -[[!meta copyright="Copyright © 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,9 +11,11 @@ is included in the section entitled [[!meta title="Microkernel FAQ"]] +See also other [[/FAQ]]. + [[!inline pages="microkernel/faq/* and !*/discussion" show=0 feeds=no actions=yes -rootpage=microkernel/faq" postformtext="Add a new item titled:"]] +rootpage="microkernel/faq" postformtext="Add a new item titled:"]] diff --git a/microkernel/fud.mdwn b/microkernel/fud.mdwn index eef829e0..3f9229aa 100644 --- a/microkernel/fud.mdwn +++ b/microkernel/fud.mdwn @@ -11,12 +11,24 @@ This article is a response to an [earlier article](http://www.linuxjournal.com/n Miles Nordin claimed that microkernels are dead already. But this is not completely true. The first generation of microkernels, which were in fact no real microkernels, are dead. But there is a new generation, which uses a radically different strategy than the original (so-called) microkernels. Thus, microkernels are still a research topic, and today they look more promising than ever before. By now, this is just something we claim, but read on, and you'll find out why we do so. -Out of our own experience, we can confirm that the first generation microkernel Mach is quite slow, but being microkernel independent is one of the goals of the Hurd and people are already working on porting the Hurd from Mach to the second generation microkernel L4. Those new second generation kernels aren't as slow as Mach and we think that one should not talk about the performance of microkernel based systems without having read at least some of the papers on L4. The L4 people did some interesting benchmarks, which indicate that one can get a lot of performance by making a microkernel really small. How is this supposed to work? Well, the microkernel provides very primitive, highly optimized operations, and applications use them to implement whichever way of interprocess communication is apropriate for them in an efficient way. By deciding this on a per-case basis, you get optimal performance for all applications. +Out of our own experience, we can confirm that the first generation microkernel +Mach is quite slow, but being microkernel independent is one of the goals of +the Hurd and people are already working on porting the Hurd from Mach to the +second generation microkernel L4. Those new second generation kernels aren't +as slow as Mach and we think that one should not talk about the performance of +microkernel based systems without having read at least some of the papers on +L4. The L4 people did some interesting benchmarks, which indicate that one can +get a lot of performance by making a microkernel really small. How is this +supposed to work? Well, the microkernel provides very primitive, highly +optimized operations, and applications use them to implement whichever way of +inter-process communication is apropriate for them in an efficient way. By +deciding this on a per-case basis, you get optimal performance for all +applications. But L4 takes this even further. For example, you can have schedulers in userspace. Therefore you can use a scheduler which is optimized for the specific tasks your system performs. With the Linux kernel, different schedulers are only possible by using a different source tree, thus you cannot switch at run-time and/or have different schedulers for different groups of processes. Of course, microkernels still have some problems, mainly because we are bound to today's technology, and current processors have not been designed with microkernels in mind. On a processor that is not optimized for systems with monolithic kernels, where the currently still problematic overhead of context switches would vanish, microkernels would get another performance boost. This sounds like an excuse, but it is intended as a reminder about the fact that the problem is not the general concept of microkernels. However, the L4 people have done a lot of good hacks to work around all this and have reached reasonable performance already. -All this could be discussed in arbitrary detail, but we won't do that now, as we have more urgent things to do than reacting on FUD about microkernels. So we will conclude by saying that it is too easy to claim that one design is fast and the other one is slow, but everything depends on how exactly a system is designed and implemented. Maybe microkernels will eventually turn out to be slower in almost any case; we doubt that, but who knows? But even then, a microkernel based system will offer enough other advantages so that people will prefer to use it in some cases. But on the other hand, history has shown that new concepts seldom replace old ones completely, but rather establish themselfes in addition to the old ones, therefore we will have the opportunity to argue about which concept is best at least for another couple of years.. or decades? +All this could be discussed in arbitrary detail, but we won't do that now, as we have more urgent things to do than reacting on FUD about microkernels. So we will conclude by saying that it is too easy to claim that one design is fast and the other one is slow, but everything depends on how exactly a system is designed and implemented. Maybe microkernels will eventually turn out to be slower in almost any case; we doubt that, but who knows? But even then, a microkernel based system will offer enough other advantages so that people will prefer to use it in some cases. But on the other hand, history has shown that new concepts seldom replace old ones completely, but rather establish themselves in addition to the old ones, therefore we will have the opportunity to argue about which concept is best at least for another couple of years.. or decades? If you are interested in research about the performance of microkernel based systems, visit <http://www.l4ka.org> and <http://os.inf.tu-dresden.de/L4/> diff --git a/microkernel/l4.mdwn b/microkernel/l4.mdwn new file mode 100644 index 00000000..7af5e6fc --- /dev/null +++ b/microkernel/l4.mdwn @@ -0,0 +1,36 @@ +[[!meta copyright="Copyright © 2004, 2006, 2007, 2008, 2010, 2011 Free Software +Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +The [*L4* microkernel](http://l4ka.org/) is an attempt to create a very small +high performace core which provides basic memory management, task and context +switching, and little else. + +[L4Ka Pistachio Home](http://l4ka.org/projects/pistachio/). + +See [l4.verified](http://nicta.com.au/research/projects/l4.verified) for work +on formally verifying an L4 microkernel. + + * {{$sel4}} + +There was a GNU/Hurd [[port to L4|history/port_to_another_microkernel]], which +is now stalled. + + +[[!ymlfront data=""" + +sel4: + + "G. Klein, K. Elphinstone, G. Heiser, J. Andronick, D. Cock, P. Derrin, + D. Elkaduwe, K. Engelhardt, R. Kolanski, M. Norrish, T. Sewell, H. Tuch, and + S. Winwood. seL4: Formal verification of an OS kernel. In Proceedings of + the ACM Symposium on OS Principles, Big Sky, MT, USA, October 2009." + +"""]] diff --git a/microkernel/mach.mdwn b/microkernel/mach.mdwn index 39d0f4d2..02627766 100644 --- a/microkernel/mach.mdwn +++ b/microkernel/mach.mdwn @@ -1,16 +1,96 @@ +[[!meta copyright="Copyright © 2007, 2008, 2010, 2012 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + Mach is a so-called first generation [[microkernel]]. It is the microkernel currently used by the [[Hurd]]. -* [[Documentation]] -* [[Concepts]] -* [[History]] ([Torvalds, Tanenbaum Debate](http://www.dina.dk/~abraham/Linus_vs_Tanenbaum.html)) + * [[Concepts]] + + * [[Deficiencies]] + + * [[Documentation]] + + * [[History]] + + * [Torvalds, Tanenbaum + Debate](http://www.dina.dk/~abraham/Linus_vs_Tanenbaum.html) + # Implementations -* [[GNU_Mach|gnumach]] -* [[Mach/OskitMach]] - A Once Successor of Mach based on OSKit -* [Apple's Darwin](http://developer.apple.com/darwin/) ([API](http://developer.apple.com/documentation/Darwin/Conceptual/KernelProgramming/index.html)) (**non-free**) + * [[GNU_Mach|gnumach]] + + * [Apple's Darwin](http://developer.apple.com/darwin/) + ([API](http://developer.apple.com/documentation/Darwin/Conceptual/KernelProgramming/index.html)) + (**non-free**) + + * [[open_issues/OSF_Mach]] + # Related -* [[Mach_Interface_Generator_(MIG)|mig]] + * [[Mach_Interface_Generator_(MIG)|mig]] + + +[[!ymlfront data=""" + +kernel_foundation_unix: + + "M. Accetta, R. Baron, W. Bolosky, D. Golub, R. Rashid, A. Tevanian, and + M. Young, Mach: A New Kernel Foundation for UNIX Development, USENIX + Conference Proceedings, July 1986. Paper + [\[pdf\]](http://www.cs.toronto.edu/~demke/469F.06/Handouts/mach_usenix86.pdf)." + +kernel_interface: + + "Mach 3 Kernel Interfaces. Open Software Foundation and Carnegie Mellon + University. Keith Loepere, Editor. NORMA-MK12: July 15, 1992. Book [\[ps + (HTTP)\]](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/kernel_interface.ps), + [\[ps + (FTP)\]](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/kernel_interface.ps)." + +kernel_principles: + + "Mach 3 Kernel Principles. Open Software Foundation and Carnegie Mellon + University. Keith Loepere. NORMA-MK12: July 15, 1992. Book [\[ps + (HTTP)\]](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/kernel_principles.ps), + [\[ps + (FTP)\]](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/kernel_principles.ps)." + +server_interface: + + "Mach 3 Server Writer’s Interfaces. Open Software Foundation and Carnegie + Mellon University. Keith Loepere, Editor. NORMA-MK12, user15: July 15, + 1992. Book [\[ps + (HTTP)\]](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/server_interface.ps), + [\[ps + (FTP)\]](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/server_interface.ps)." + +server_writer: + + "Mach 3 Server Writer’s Guide. Open Software Foundation and Carnegie Mellon + University. Keith Loepere, Editor. NORMA-MK12, user15: July 15, 1992. Book + [\[ps + (HTTP)\]](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/server_writer.ps), + [\[ps + (FTP)\]](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/server_writer.ps)." + +vm: + + "R. Rashid, A. Tevanian, M. Young, D. Golub, and R. Baron, + Machine-Independent Virtual Memory Management for Paged Uniprocessor and + Multiprocessor Architectures, 2nd ACM Symposium on Architectural Support for + Programming Languages and Operating Systems (ASPLOS), October 1987. Paper + [\[pdf\]](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.111.7918&rep=rep1&type=pdf), + presentation + [\[ppt\]](http://www2.cs.uh.edu/~paris/6360/PowerPoint/Mach.ppt)." + +"""]] diff --git a/microkernel/mach/concepts.mdwn b/microkernel/mach/concepts.mdwn index 04dbb1c6..0f7cbf00 100644 --- a/microkernel/mach/concepts.mdwn +++ b/microkernel/mach/concepts.mdwn @@ -1,6 +1,33 @@ -[[Mach]] is a first-generation [[microkernel]]. Mach's basic abstractions -include [[address_space]]s in the form of [[task]]s, execution contexts in the -form of [[thread]]s, [[IPC]], [[capabilities|capability]] in the form of [[port]]s, and -[[memory_object]]s, which enable Mach's [[external_pager_mechanism]]. +[[!meta copyright="Copyright © 2002, 2003, 2007, 2010 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[Mach]] is a first-generation [[microkernel]]. + +Mach's basic abstractions include [[virtual_address_space]]s in the form of +[[task]]s, execution contexts in the form of [[thread]]s, [[IPC]], +[[capabilities|capability]] in the form of [[port]]s, and [[memory_object]]s, +which enable Mach's [[external_pager_mechanism]]. + +Controlling [[task]]s, their [[virtual_address_space]], [[thread]]s, and other +system objects in Mach is implemented by using [[port]]s, as opposed to other +[[kernel]]s' [[system_call]] interface: almost all of the Mach API is +implemented by sending [[message]]s to [[port]]s. Device drivers that reside +in kernel space are controlled by ports, too. Mach's [[API]] is well-[[documented|documentation]]. + +[[!toggleable id=mach_kernel_principles text="""[[!template id=note +text="*[[mach\_kernel\_principles|documentation]]*: +{{$mach#kernel_principles}}"]]"""]] + +In particular the [[!toggle id=mach_kernel_principles +text="[mach\_kernel\_principles]"]] book further elaborates on Mach's concepts +and principles. diff --git a/microkernel/mach/continuation.mdwn b/microkernel/mach/continuation.mdwn new file mode 100644 index 00000000..7a3267f3 --- /dev/null +++ b/microkernel/mach/continuation.mdwn @@ -0,0 +1,24 @@ +[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[Mach]] internally uses *continuation*s for kernel [[thread]] management. + +The advantage is that not a full kernel thread stack has to be preserved in +case that a thread is about to enter a blocking state. This saves space. It +is not clear this is still worthwhile given today's RAM offerings. (How many +kernel threads are there, typically?) + +And, this would no longer be possible in case Mach were be made a +[[preemptive|preemtion]] kernel. In the latter case, the kernel itself, that +is, kernel threads can be preempted, and then their full state needs to be +preserved. + +[[!tag open_issue_documentation]] <!-- Not linked to from any Mach page. Move +to GNU Mach pages, as this is only an implementation detail? --> diff --git a/microkernel/mach/deficiencies.mdwn b/microkernel/mach/deficiencies.mdwn new file mode 100644 index 00000000..f2f49975 --- /dev/null +++ b/microkernel/mach/deficiencies.mdwn @@ -0,0 +1,260 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation open_issue_gnumach]] + + +# IRC, freenode, #hurd, 2012-06-29 + + <henrikcozza> I do not understand what are the deficiencies of Mach, the + content I find on this is vague... + <antrik> the major problems are that the IPC architecture offers poor + performance; and that resource usage can not be properly accounted to the + right parties + <braunr> antrik: the more i study it, the more i think ipc isn't the + problem when it comes to performance, not directly + <braunr> i mean, the implementation is a bit heavy, yes, but it's fine + <braunr> the problems are resource accounting/scheduling and still too much + stuff inside kernel space + <braunr> and with a very good implementation, the performance problem would + come from crossing address spaces + <braunr> (and even more on SMP, i've been thinking about it lately, since + it would require syncing mmu state on each processor currently using an + address space being modified) + <antrik> braunr: the problem with Mach IPC is that it requires too many + indirections to ever be performant AIUI + <braunr> antrik: can you mention them ? + <antrik> the semantics are generally quite complex, compared to Coyotos for + example, or even Viengoos + <braunr> antrik: the semantics are related to the message format, which can + be simplified + <braunr> i think everybody agrees on that + <braunr> i'm more interested in the indirections + <antrik> but then it's not Mach IPC anymore :-) + <braunr> right + <braunr> 22:03 < braunr> i mean, the implementation is a bit heavy, yes, + but it's fine + <antrik> that's not an implementation issue + <braunr> that's what i meant by heavy :) + <braunr> well, yes and no + <braunr> Mach IPC have changed over time + <braunr> it would be newer Mach IPC ... :) + <antrik> the fact that data types are (supposed to be) transparent to the + kernel is a major part of the concept, not just an implementation detail + <antrik> but it's not just the message format + <braunr> transparent ? + <braunr> but they're not :/ + <antrik> the option to buffer in the kernel also adds a lot of complexity + <braunr> buffer in the kernel ? + <braunr> ah you mean message queues + <braunr> yes + <antrik> braunr: eh? the kernel parses all the type headers during transfer + <braunr> yes, so it's not transparent at all + <antrik> maybe you have a different understanding of "transparent" ;-) + <braunr> i guess + <antrik> I think most of the other complex semantics are kinda related to + the in-kernel buffering... + <braunr> i fail to see why :/ + <antrik> well, it allows ports rights to be destroyed while a message is in + transfer. a lot of semantics revolve around what happens in that case + <braunr> yes but it doesn't affect performance a lot + <antrik> sure it does. it requires a lot of extra code and indirections + <braunr> not a lot of it + <antrik> "a lot" is quite a relative term :-) + <antrik> compared to L4 for example, it *is* a lot + <braunr> and those indirections (i think you refer to more branching here) + are taken only when appropriate, and can be isolated, improved through + locality, etc.. + <braunr> the features they add are also huge + <braunr> L4 is clearly insufficient + <braunr> all current L4 forks have added capabilities .. + <braunr> (that, with the formal verification, make se4L one of the + "hottest" recent system projects) + <braunr> seL4* + <antrik> yes, but with very few extra indirection I think... similar to + EROS (which claims to have IPC almost as efficient as the original L4) + <braunr> possibly + <antrik> I still fail to see much real benefit in formal verification :-) + <braunr> but compared to other problems, this added code is negligible + <braunr> antrik: for a microkernel, me too :/ + <braunr> the kernel is already so small you can simply audit it :) + <antrik> no, it's not neglible, if you go from say two cache lines touched + per IPC (original L4) to dozens (Mach) + <antrik> every additional variable that needs to be touched to resolve some + indirection, check some condition adds significant overhead + <braunr> if you compare the dozens to the huge amount of inter processor + interrupt you get each time you change the kernel map, it's next to + nothing .. + <antrik> change the kernel map? not sure what you mean + <braunr> syncing address spaces on hundreds of processors each time you + send a message is a real scalability issue here (as an example), where + Mach to L4 IPC seem like microoptimization + <youpi> braunr: modify, you mean? + <braunr> yes + <youpi> (not switchp + <youpi> ) + <braunr> but that's only one example + <braunr> yes, modify, not switch + <braunr> also, we could easily get rid of the ihash library + <braunr> making the message provide the address of the object associated to + a receive right + <braunr> so the only real indirection is the capability, like in other + systems, and yes, buffering adds a bit of complexity + <braunr> there are other optimizations that could be made in mach, like + merging structures to improve locality + <pinotree> "locality"? + <braunr> having rights close to their target port when there are only a few + <braunr> pinotree: locality of reference + <youpi> for cache efficiency + <antrik> hundreds of processors? let's stay realistic here :-) + <braunr> i am .. + <braunr> a microkernel based system is also a very good environment for RCU + <braunr> (i yet have to understand how liburcu actually works on linux) + <antrik> I'm not interested in systems for supercomputers. and I doubt + desktop machines will get that many independant cores any time soon. we + still lack software that could even romotely exploit that + <braunr> hum, the glibc build system ? :> + <braunr> lol + <youpi> we have done a survey over the nix linux distribution + <youpi> quite few packages actually benefit from a lot of cores + <youpi> and we already know them :) + <braunr> what i'm trying to say is that, whenever i think or even measure + system performance, both of the hurd and others, i never actually see the + IPC as being the real performance problem + <braunr> there are many other sources of overhead to overcome before + getting to IPC + <youpi> I completely agree + <braunr> and with the advent of SMP, it's even more important to focus on + contention + <antrik> (also, 8 cores aren't exactly a lot...) + <youpi> antrik: s/8/7/ , or even 6 ;) + <antrik> braunr: it depends a lot on the use case. most of the problems we + see in the Hurd are probably not directly related to IPC performance; but + I pretty sure some are + <antrik> (such as X being hardly usable with UNIX domain sockets) + <braunr> antrik: these have more to do with the way mach blocks than IPC + itself + <braunr> similar to the ext2 "sleep storm" + <antrik> a lot of overhead comes from managing ports (for for example), + which also mostly comes down to IPC performance + <braunr> antrik: yes, that's the main indirection + <braunr> antrik: but you need such management, and the related semantics in + the kernel interface + <braunr> (although i wonder if those should be moved away from the message + passing call) + <antrik> you mean a different interface for kernel calls than for IPC to + other processes? that would break transparency in a major way. not sure + we really want that... + <braunr> antrik: no + <braunr> antrik: i mean calls specific to right management + <antrik> admittedly, transparency for port management is only useful in + special cases such as rpctrace, and that probably could be served better + with dedicated debugging interfaces... + <braunr> antrik: i.e. not passing rights inside messages + <antrik> passing rights inside messages is quite essential for a capability + system. the problem with Mach IPC in regard to that is that the message + format allows way more flexibility than necessary in that regard... + <braunr> antrik: right + <braunr> antrik: i don't understand why passing rights inside messages is + important though + <braunr> antrik: essential even + <youpi> braunr: I guess he means you need at least one way to pass rights + <antrik> braunr: well, for one, you need to pass a reply port with each RPC + request... + <braunr> youpi: well, as he put, the message passing call is overpowered, + and this leads to many branches in the code + <braunr> antrik: the reply port is obvious, and can be optimized + <braunr> antrik: but the case i worry about is passing references to + objects between tasks + <braunr> antrik: rights and identities with the auth server for example + <braunr> antrik: well ok forget it, i just recall how it actually works :) + <braunr> antrik: don't forget we lack thread migration + <braunr> antrik: you may not think it's important, but to me, it's a major + improvement for RPC performance + <antrik> braunr: how can seL4 be the most interesting microkernel + then?... ;-) + <braunr> antrik: hm i don't know the details, but if it lacks thread + migration, something is wrong :p + <braunr> antrik: they should work on viengoos :) + <antrik> (BTW, AIUI thread migration is quite related to passive objects -- + something Hurd folks never dared seriously consider...) + <braunr> i still don't know what passive objects are, or i have forgotten + it :/ + <antrik> no own control threads + <braunr> hm, i'm still missing something + <braunr> what do you refer to by control thread ? + <braunr> with* + <antrik> i.e. no main loop etc.; only activated by incoming calls + <braunr> ok + <braunr> well, if i'm right, thomas bushnel himself wrote (recently) that + the ext2 "sleep" performance issue was expected to be solved with thread + migration + <braunr> so i guess they definitely considered having it + <antrik> braunr: don't know what the "sleep peformance issue" is... + <braunr> http://lists.gnu.org/archive/html/bug-hurd/2011-12/msg00032.html + <braunr> antrik: also, the last message in the thread, + http://lists.gnu.org/archive/html/bug-hurd/2011-12/msg00050.html + <braunr> antrik: do you consider having a reply port being an avoidable + overhead ? + <antrik> braunr: not sure. I don't remember hearing of any capability + system doing this kind of optimisation though; so I guess there are + reasons for that... + <braunr> antrik: yes me too, even more since neal talked about it on + viengoos + <antrik> I wonder whether thread management is also such a large overhead + with fully sync IPC, on L4 or EROS for example... + <braunr> antrik: it's still a very handy optimization for thread scheduling + <braunr> antrik: it makes solving priority inversions a lot easier + <antrik> actually, is thread scheduling a problem at all with a thread + activation approach like in Viengoos? + <braunr> antrik: thread activation is part of thread migration + <braunr> antrik: actually, i'd say they both refer to the same thing + <antrik> err... scheduler activation was the term I wanted to use + <braunr> same + <braunr> well + <braunr> scheduler activation is too vague to assert that + <braunr> antrik: do you refer to scheduler activations as described in + http://en.wikipedia.org/wiki/Scheduler_activations ? + <antrik> my understanding was that Viengoos still has traditional threads; + they just can get scheduled directly on incoming IPC + <antrik> braunr: that Wikipedia article is strange. it seems to use + "scheduler activations" as a synonym for N:M multithreading, which is not + at all how I understood it + <youpi> antrik: I used to try to keep a look at those pages, to fix such + wrong things, but left it + <braunr> antrik: that's why i ask + <antrik> IIRC Viengoos has a thread associated with each receive + buffer. after copying the message, the kernel would activate the + processes activation handler, which in turn could decide to directly + schedule the thead associated with the buffer + <antrik> or something along these lines + <braunr> antrik: that's similar to mach handoff + <youpi> antrik: generally enough, all the thread-related pages on wikipedia + are quite bogus + <antrik> nah, handoff just schedules the process; which is not useful, if + the right thread isn't activated in turn... + <braunr> antrik: but i think it's more than that, even in viengoos + <youpi> for instance, the french "thread" page was basically saying that + they were invented for GUIs to overlap computation with user interaction + <braunr> .. :) + <antrik> youpi: good to know... + <braunr> antrik: the "misunderstanding" comes from the fact that scheduler + activations is the way N:M threading was implemented on netbsd + <antrik> youpi: that's a refreshing take on the matter... ;-) + <braunr> antrik: i'll read the critique and viengoos doc/source again to be + sure about what we're talking :) + <braunr> antrik: as threading is a major issue in mach, and one of the + things i completely changed (and intend to change) in x15, whenever i get + to work on that again ..... :) + <braunr> antrik: interestingly, the paper about scheduler activations was + written (among others) by brian bershad, in 92, when he was actively + working on research around mach + <antrik> braunr: BTW, I have little doubt that making RPC first-class would + solve a number of problems... I just wonder how many others it would open diff --git a/microkernel/mach/discussion.mdwn b/microkernel/mach/discussion.mdwn deleted file mode 100644 index 589e302d..00000000 --- a/microkernel/mach/discussion.mdwn +++ /dev/null @@ -1,23 +0,0 @@ -## <a name="Maintenance_of_the_Mach_web"> Maintenance of the Mach web </a> - -**_Old discussions:_** [[WIKIHOMEURLMachTOPICrev13]] - -Interesting, for consistency sake I'll think about making your changes you made on the right hand side to the other web WebHome pages. I guess it's not critical that they are identical, but I was trying to keep them identical if possible. I also wanted it to be "light" enough feature wise that it doesn't overpower the page. You've added back a few of the features, so we obviously differ in how important you and I think these features are. That's OK, I'll think about it some more and we'll see what happens. - -Oh, I see you added back [[WebTopicList]] and [[WebPreferences]]. I purposely removed [[WebPreferences]] from the lists on the right because it has nothing to do with navigation. I also didn't think that people actually use topic names to navigate. If they do they could search for them. Keeping the number to four items instead of six and keeping the descriptions concise makes a big difference when I view the page. - -(goes off to think more...) - -and eat... ;-) - --- [[Main/GrantBow]] - 29 Dec 2002 - -**_Reasons for my change:_** - -1. [[WebTopicList]] is a lot quicker than the [[WebIndex]] - brings down the load times and the load of the server -2. [[WebPreferences]] - users might be curious to see what can be modified. Changes should of course only be made in their home topics, like in %WIKIUSERNAME%. However, the [[WebPreferences]] can serve as an inspiration. Therefore we should perhaps make sure only the [[Main/TWikiAdminGroup]] members can alter the \*Preferences topics. -3. If you look closely I've also reordered the links. Shorter names first and long ones last, I tried to keep the descriptions brief and in proportional length as well. - -I don't know about you, but keeping the number of items to four rather than six doesn't really matter to me. The text is quite small and if it's the space we're after the [[WebStatistics]] does take up more than the navigation links. - --- [[Main/JoachimNilsson]] - 29 Dec 2002 diff --git a/microkernel/mach/documentation.mdwn b/microkernel/mach/documentation.mdwn index 3b12bfac..cc880ab6 100644 --- a/microkernel/mach/documentation.mdwn +++ b/microkernel/mach/documentation.mdwn @@ -1,24 +1,33 @@ -[[!meta copyright="Copyright © 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008 -Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, +2010 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] - - [Meet Mach](http://www.stepwise.com/Articles/Technical/MeetMach.html), a - summary of Mach's history and main concepts. + * Mach's [[concepts]]. + + * [*Meet Mach* by James + Scott](http://beefchunk.com/documentation/macosx-programming/Meet_Mach.pdf), + a summary of Mach's history and main concepts. * *[[The_GNU_Mach_Reference_Manual|gnumach/reference_manual]]*. - - OSF's [Kernel Interface (ps)](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/kernel_interface.ps) - [Kernel Interface (pdf)](http://shakthimaan.com/downloads/hurd/kernel_interface.pdf) + * {{$mach#kernel_foundation_unix}} + + * {{$mach#vm}} + + * {{$mach#kernel_principles}} + + * {{$mach#kernel_interface}} + + * {{$mach#server_writer}} - - OSF's [Kernel Principles (ps)](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/kernel_principles.ps) - [Kernel Principles (pdf)](http://shakthimaan.com/downloads/hurd/kernel_principles.pdf) + * {{$mach#server_interface}} * [*The Unofficial GNU Mach IPC beginner's guide*](http://hurdextras.nongnu.org/ipc_guide/), an easy introduction to diff --git a/microkernel/mach/external_pager_mechanism.mdwn b/microkernel/mach/external_pager_mechanism.mdwn index b175d1cc..05a6cc56 100644 --- a/microkernel/mach/external_pager_mechanism.mdwn +++ b/microkernel/mach/external_pager_mechanism.mdwn @@ -1,92 +1,93 @@ -[[!meta copyright="Copyright © 2002, 2007, 2008 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2002, 2007, 2008, 2010, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -Mach provides a so-called external pager [[mechanism]]. This +Mach provides a so-called *external pager [[mechanism]]*. This mechanism serves to separate *managing memory* from *managing -content*. Mach does the former while user space tasks do the +content*. Mach does the former while user-space processes do the latter. +[[!tag open_issue_documentation]] <!-- Should probably refer to {{$mach#vm}}. +--> + # Introduction -In Mach, a task's [[Mach/AddressSpace]] consists of references -to [[Mach/MemoryObjects]]. A memory object is designated using -a [[port]] (a port is just a [[capability]]) and -implemented by a normal process. +In Mach, a [[task]]'s [[virtual_address_space]] consists of references to +[[memory_object]]s. To associate a memory object with a portion of a task's -address space, vm\_map is invoked a capability designating +address space, `vm_map` is invoked on a capability designating the task and passing a reference to the memory object and the offset at which to install it. (The first time a task maps an object, Mach sends an initialization message to the server including a control capability, which it uses to supply pages to the kernel.) This is essentially -the same as mapping a file into an address space on [[Unix]] -using mmap. +the same as mapping a file into an address space on [[UNIX]] +using `mmap`. -When a task faults, Mach checks to see if there is a memory +When a task [[faults|page_fault]], Mach checks to see if there is a memory object associated with the fault address. If not, the task -is sent an exception, which is normally further propagated +is sent an [[exception]], which is normally further propagated as a segmentation fault. If there is an associated memory -object, Mach checks whether the corresponding page is in core. -If it is, it installs the page and resumes the task. Mach -then invokes the memory object with the memory\_object\_request +object, Mach checks whether the corresponding [[page]] is in core. +If it is, it installs the page and resumes the task. Mach +then invokes the memory object with the `memory_object_request` method and the page to read. The memory manager then fetches or creates the content as appropriate and supplies it to -Mach using the memory\_object\_supply method. +Mach using the `memory_object_supply` method. # Creating and Mapping a Memory Object The following illustrates the basic idea: -> ________ -> / \ -> | Mach | -> \________/ -> /| / |\ \ -> (C) vm_map / / m_o_ready (E)\ \ (D) memory_object_init -> / |/ (F) return \ \| -> ________ ________ -> / \ -----> / \ -> | Client | (A) open | Server | -> \________/ <----- \________/ -> (B) memory_object - -(A) The client sends an "open" rpc to the server. + ________ + / \ + | Mach | + \________/ + /| / |\ \ + (C) vm_map / / m_o_ready (E)\ \ (D) memory_object_init + / |/ (F) return \ \| + ________ ________ + / \ -----> / \ + | Client | (A) open | Server | + \________/ <----- \________/ + (B) memory_object + +(A) The client sends an `open` [[RPC]] to the server. (B) The server creates a memory object (i.e., a port receive right), adds it to the port set that it is listening on and returns a capability (a port send right) to the client. (C) The client attempts to map the object into its address space using -the vm\_map rpc. It passes a reference to the port that the server gave +the `vm_map` RPC. It passes a reference to the port that the server gave it to the vm server (typically Mach). (D) Since Mach has never seen the object before, it queues a -memory\_object\_init on the given port along with a send right (the +`memory_object_init` on the given port along with a send right (the memory control port) for the manager to use to send messages to the kernel and also as an authentication mechanism for future interactions: the port is supplied so that the manager will be able to -identify from which kernel a given memory\_object\_* IPC is from. +identify from which kernel a given `memory_object_*` IPC is from. (E) The server dequeues the message, initializes internal data structures to manage the mapping and then invokes the -memory\_object\_ready method on the control object. +`memory_object_ready` method on the control object. (F) The kernel sees that the manager is ready, sets up the appropriate -mappings in the client and then replies to the vm\_map rpc indicating +mappings in the client's address space and then replies to the `vm_map` RPC indicating success. -There is nothing stopping others from playing "the kernel." This is +There is nothing stopping others from playing *the kernel*. This is not a security problem: clients must [[trust]] the server from whom they obtain memory objects and also the servers with whom they share the object. Multiple memory managers are a reality that should be @@ -96,37 +97,37 @@ mappings etc. # Resolving Page Faults -> (G) Client ________ -> resumed / \ -> | Mach | -> (A) Fault +----|------+ | \ (B) m_o_request (C) store_read -> ____|___ \_____|__/ |\ \| ________ _________ -> / +---\-------+ \ / \ / \ -> | Client | (F) | Server |<===>| storeio | -> \________/ m_o_supply \________/ \_________/ -> (E) return data | ^ -> | | (D) device_read -> v | -> ________ -> / Device \ -> | Driver | -> \________/ -> | ^ -> | | -> v -> ____________ -> / Hardware \ - -(A) The client does a memory access and faults. The kernel catches + (G) Client ________ + resumed / \ + | Mach | + (A) Fault +----|------+ | \ (B) m_o_request (C) store_read + ____|___ \_____|__/ |\ \| ________ _________ + / +---\-------+ \ / \ / \ + | Client | (F) | Server |<===>| storeio | + \________/ m_o_supply \________/ \_________/ + (E) return data | ^ + | | (D) device_read + v | + ________ + / Device \ + | Driver | + \________/ + | ^ + | | + v + ____________ + / Hardware \ + +(A) The client does a memory access and [[faults|page_fault]]. The kernel catches the fault and maps the address to the appropriate memory object. It -then invokes the memory\_object\_request method on the associated +then invokes the `memory_object_request` method on the associated capability. (In addition to the page to supply, it also supplies the control port so that the server can determine which kernel sent the message.) -(B) The manager dequeues the message. On the Hurd, this is translated -into a store\_read: a function in the libstore library which is used to -transparently manage block devices. The storeio server starts off as +(B) The manager dequeues the message. On the [[Hurd]], this is translated +into a `store_read`: a function in the [[hurd/libstore]] library which is used to +transparently manage block devices. The [[hurd/storeio]] server starts off as a separate process, however, if the server has the appropriate permission, the backing object can be contacted directly by the server. This layer of indirection is desirable when, for instance, a @@ -134,37 +135,37 @@ storeio running as root may want to only permit read only access to a resource, yet it cannot safely transfer its handle to the client. In this case, it would proxy the requests. -(C) The storeio server contacts, for instance, a device driver to do +(C) The storeio server contacts, for instance, a [[device_driver]] to do the read. This could also be a network block device (the NBD server in GNU/Linux), a file, a memory object, etc. -(D) The device driver allocates an anonymous page from the default -pager and reads the data into it. Once all of the operations are +(D) The device driver allocates an [[anonymous_page]] from the +[[default_pager]] and reads the data into it. Once all of the operations are complete, the device returns the data to the client unmapping it from its own address space at the same time. -(E) The storeio transfers the page to the server. The page is still +(E) The storeio server transfers the page to the server. The page is still anonymous. -(F) The manager does a memory\_object\_supply transferring the page to +(F) The manager does a `memory_object_supply` transferring the page to the kernel. Only now is the page not considered to be anonymous but managed. (G) The kernel caches the page, installs it in the client's virtual -address space and finally, resumes the client. +[[address_space]] and finally, resumes the client. # Paging Data Out -> Change manager Pager m_o_return store_write -> \ _________ (B) __(A)__ (C) ________ (D) _______ -> S | / Default \ / \ / \ / \ -> W |<=>| Pager |<=>| Mach |==>| server |<=>| storeio |<=> -> A | \_________/ \________/ \________/ \_______/ -> P | -> / + Change manager Pager m_o_return store_write + \ _________ (B) __(A)__ (C) ________ (D) _______ + S | / Default \ / \ / \ / \ + W |<=>| Pager |<=>| Mach |==>| server |<=>| storeio |<=> + A | \_________/ \________/ \________/ \_______/ + P | + / -(A) The paging [[policy]] is implemented by Mach: servers just implement +(A) The [[paging]] [[policy]] is implemented by Mach: servers just implement the [[mechanism]]. (B) Once the kernel has selected a page that it would like to evict, it @@ -173,10 +174,22 @@ if the server does not deallocate the page quickly enough, it cannot cause a denial of service: the kernel will just later double page it to swap (the default pager is part of the [[tcb]]). -(C) Mach then invokes memory\_object\_return method on the control -object. The server is expected to save the page free it in a timely +(C) Mach then invokes `memory_object_return` <!-- doesn't exist --> method on the control +object. The server is expected to save the page free <!-- ? --> it in a timely fashion. The server is not required to send a response to the kernel. -(D) The manager then transfers the data to the storeio which +(D) The manager then transfers the data to the storeio server which eventually sends it to disk. The device driver consumes the memory -doing the equivalent of a vm\_deallocate. +doing the equivalent of a `vm_deallocate`. + + +# Issues + + * [[open_issues/performance/io_system/read-ahead]] + + * [[open_issues/performance/io_system/clustered_page_faults]] + + +# GNU Hurd Usage + +Read about the [[Hurd's I/O path|hurd/io_path]]. diff --git a/microkernel/mach/gnumach.mdwn b/microkernel/mach/gnumach.mdwn index f3d6d5f9..edd0cfdb 100644 --- a/microkernel/mach/gnumach.mdwn +++ b/microkernel/mach/gnumach.mdwn @@ -1,15 +1,18 @@ -[[!meta copyright="Copyright © 2001, 2002, 2007, 2008 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2001, 2002, 2007, 2008, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -GNU Mach is the microkernel that the [[GNU_Hurd|hurd]] system is based on. +GNU Mach is the microkernel upon which a GNU Hurd system is based. It provides +an Inter Process Communication (IPC) mechanism that the Hurd uses to define +interfaces for implementing in a distributed multi-server fashion the services +a traditional operating system kernel provides. It is maintained by the Hurd developers for the GNU project and remains compatible with [[Mach]] 3.0. @@ -75,6 +78,7 @@ GNU/Hurd. * [[Building]] * [[Debugging]] * [[Boot_Trace]] + * [[Memory_Management]] * [[Projects]] * [[Rules]] * [[Open Issues|tag/open_issue_gnumach]] diff --git a/microkernel/mach/gnumach/boot_trace.mdwn b/microkernel/mach/gnumach/boot_trace.mdwn index d33ef25a..1badf712 100644 --- a/microkernel/mach/gnumach/boot_trace.mdwn +++ b/microkernel/mach/gnumach/boot_trace.mdwn @@ -1,12 +1,13 @@ -[[!meta copyright="Copyright © 2007, 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2007, 2008, 2011 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] `if NCPUS > 1` stuff is not being considered so far. @@ -215,6 +216,12 @@ is included in the section entitled >> kern/bootstrap.c: bootstrap\_create +>>> The [[grub/multiboot]] modules have been put somewhere into memory by +>>> [[GRUB]]. The boot scripts are parsed. The modules' ELF image's `PT_LOAD` +>>> sections are \`\`read'' (that is, `vm_allocate` and `copyout`) and turned +>>> into real [[task]]s. The multiboot modules' memory regions can be +>>> deallocated then. + >> [...] >> vm\_pageout diff --git a/microkernel/mach/gnumach/building.mdwn b/microkernel/mach/gnumach/building.mdwn index 9c075600..427fb083 100644 --- a/microkernel/mach/gnumach/building.mdwn +++ b/microkernel/mach/gnumach/building.mdwn @@ -1,5 +1,13 @@ -Additional to the following text, a further [[example]] has be posted. +[[!meta copyright="Copyright © 2006, 2007, 2008, 2011 Free Software Foundation, +Inc."]] +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] # Building [[GNU_Mach|gnumach]] from Source @@ -11,55 +19,32 @@ enabled) is around 50 MiB. ## Getting the Source Code -### Developers's RCS +You can either use the git repository (see <http://git.savannah.gnu.org/cgit/hurd/>), -See <http://savannah.gnu.org/cvs/?group=hurd>. + $ git clone http://git.savannah.gnu.org/cgit/hurd/gnumach.git/ - $ cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/sources/hurd co -r gnumach-1-branch gnumach - -(Most probably you want to get hold of the *GNU Mach 1 branch* and not the -trunk, which is also what we've done above.) - -You then have to create the automatically generatable files: - - $ ( cd gnumach && autoreconf --install ) - -### What Debian is currently using - -See [here](http://packages.debian.net/source/unstable/gnumach). +... or get the Debian sources, if you're using Debian. (See +[here](http://packages.debian.net/source/unstable/gnumach).) $ apt-get source gnumach Please see the Debian [[running/debian/FAQ]] before using `apt-get source`. -## Preparing for the Build +## On Debian Systems: -### ... on Debian systems +### Preparing for the Build -Building GNU Mach requires the *build-essential* and *fakeroot* packages, their -dependencies and additional packages that are specified by the source gnumach -package: +Building GNU Mach requires the *build-essential* and *fakeroot* packages, +and some additional dependencies specified by the gnumach source package: # apt-get install build-essential fakeroot # apt-get build-dep gnumach -### ... on non-Debian systems - -Apart from the case that you only want to install GNU Mach's header files (see -below), building GNU Mach requires you to have the Mach Interface Generator -installed. See [[building_MIG|mig/gnu_mig/building]] about how to do that, then come -back here. +### Building and Installing ... Debian `.deb` files -Additionally, building GNU Mach requires a C compiler, a standard C library and -your favourite flavor of awk (gawk) and GNU make. +Change into the directory with the downloaded / unpacked GNU Mach sources, -## Building and Installing - -### ... Debian `.deb` files - -Change into the directory with the downloaded / unpacked GNU Mach sources, e.g. - - $ cd gnumach-20050801 + $ cd gnumach-XXXXXXXX Start the build process with @@ -67,48 +52,68 @@ Start the build process with [[GNU_Mach|gnumach]] is now building. To use the new kernel, you must install the resulting `.deb` package which is located one directory above the build -directory and has a similar name as the build directory, e.g. +directory and has a similar name as the build directory: - # dpkg -i ../gnumach_20050801-4_hurd-i386.deb + # dpkg -i ../gnumach_XXXXXXXX-X_hurd-i386.deb You can now reboot your computer and enjoy the new kernel. -### [TODO] +## On non-Debian Systems: -GNU Mach should be built in a separate directory: +### Preparing for the Build - $ mkdir gnumach-build - $ cd gnumach-build +Building GNU Mach requires a C compiler, a _static_ 32 bit standard C library, +your favourite flavor of awk (gawk) and GNU make. -Find the path to your GNU Mach sources (`[...]/gnumach-1-branch`) and configure -it: +First, create the configuartion files: - $ [...]/gnumach-1-branch/configure [TODO] + $ cd gnumach + $ autoreconf --install -Build the kernel image: +GNU Mach (and the associated headers) need be built in a separate build directory: - $ make gnumach.gz + $ mkdir build + $ cd build -Optionally run the (tiny) test suite: +Run configure: - $ make check + $ ../configure --prefix= -You can then install and use `gnumach.gz`. +If building on a 64 bit host system, +you need a number of additional settings to force a 32 bit build: -[TODO.] + $ CPP='gcc -m32 -E -x c -undef -ansi' CC='gcc -m32' LD='ld -melf_i386' ../configure --prefix= --host=i686-unknown-linux-gnu -### Installing only the Header Files +### Installing the Header Files First -GNU Mach should be built in a separate directory: +In order to build GNU Mach, you will need a working MIG. +Building MIG in turn requires the GNU Mach header files to be already present. +So for bootstrapping MIG, you have to install the Mach headers first, +for example into `~/gnu/include/`: - $ mkdir gnumach-build - $ cd gnumach-build + $ make DESTDIR=~/gnu install-data -Find the path to your GNU Mach sources (`[...]/gnumach-1-branch`) and configure -it: +Now you can [[build_MIG|mig/gnu_mig/building]]. +Once you are done with that, come back here to finish the Mach build. - $ [...]/gnumach-1-branch/configure --prefix= +### Building and Installing -Install the header files into e.g. `~/gnu/include/`: +With MIG present, now build the kernel image: + + $ make gnumach.gz + +Optionally run the (tiny) test suite: + + $ make check + +It's a good idea to make a backup of the previously installed kernel, in case +you can't boot using the new one. That way, you can restore it after booting +from a rescue media (or mounting the disk image used by your vm). + + # cp /boot/gnumach.gz /boot/gnumach.gz.bak + +GNU Mach can now be moved into place, typically `/boot/gnumach.gz`, so that you +can boot your system with the new kernel. + + # cp gnumach.gz /boot - $ make DESTDIR=~/gnu install-data diff --git a/microkernel/mach/gnumach/building/example.mdwn b/microkernel/mach/gnumach/building/example.mdwn deleted file mode 100644 index 7db98547..00000000 --- a/microkernel/mach/gnumach/building/example.mdwn +++ /dev/null @@ -1,54 +0,0 @@ -[[!meta copyright="Copyright © 2007, 2008 Free Software Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] - -## Compiling GNU Mach microkernel - -Host development system is IBM T41 running Debian Sarge 3.1r0a GNU/Linux. - -* gcc version: 3.3.5 -* GNU sed version: 4.1.2 -* GNU make version: 3.8 -* mig version: 1.3-4 - -Obtained gnumach-1-branch sources from cvs: - - export CVS_RSH="ssh" - cvs -z3 -d:ext:anoncvs@ savannah.gnu.org:/cvsroot/hurd co -r gnumach-1-branch gnumach - -Obtained mig_1.3-4_i386.deb from -http://www.hadrons.org/~guillem/debian/pool/main/mig/. Installed it using dpkg: - - dpkg -i mig_1.3-4_i386.deb - -Entered into the gnumach sources and did the following for compilation: - - mkdir build - cd build - ../configure --host=i386-unknown-gnu0.2 --build=i586-pc-linux-gnu \ - --enable-kdb --enable-ide - make - -The kernel file is created in the build directory. Move it to /boot on the -testing x86 system Hurd partition. Rename it as gnumach1 and compress it: - - mv kernel gnumach1 - gzip gnumach1 - -Add a new entry on the testing machine /boot/grub/menu.lst to boot the new -kernel. - - title GNU Hurd K10 Compiled gnumach - kernel (hd0,3)/boot/gnumach1.gz root=device:hd2s4 -s - module (hd0,3)/hurd/ext2fs.static--multiboot-command-line=${kernel-command-line} \\ - --host-priv-port=${host-port} --device-master-port=${device-port} \\ - --exec-server-task=${exec-task} -T typed ${root} $(task-create)$(task-resume) - module (hd0,3)/lib/ld.so.1 /hurd/exec $(exec-task=task-create) - -Reboot into the new compiled mygnumach1.gz kernel! diff --git a/microkernel/mach/gnumach/debugging.mdwn b/microkernel/mach/gnumach/debugging.mdwn index 3a93c6ad..71e92459 100644 --- a/microkernel/mach/gnumach/debugging.mdwn +++ b/microkernel/mach/gnumach/debugging.mdwn @@ -1,23 +1,82 @@ -[[!meta copyright="Copyright © 2007, 2008, 2009 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2007, 2008, 2009, 2011, 2012 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +Here are some hints to debug with GNU Mach. + +[[!toc levels=2]] + + +# Kernel Debugger Mach has a built-in kernel debugger. [Manual](http://www.gnu.org/software/hurd/gnumach-doc/Kernel-Debugger.html). +First, make sure to enable it. Either by using a pre-packaged gnumach-image-something-dbg, or by passing --enable-kdb to the ./configure invocation. + +Then, reproduce the issue again. If something like a kernel trap happens, you will end up in the GNU Mach debugger. Otherwise, type control-alt-d to make Mach enter it by hand. + +If you are running in kvm or qemu, it is convenient to use the curses frontend to be able to copy/paste. + +To get the register values, type + + show registers + +To get a backtrace, type + + trace + +, which will print both function return addresses and function parameters, such as + + 0x107cf1(8088488,5e,40000008,2aa008,0) + 0x1071bc(0,0,0,0,0) + 0x106831(24fe00,2000,b,800,0) + +Run the addr2line tool on the return addresses: + + $ addr2line -i -f -e /boot/gnumach 0x107cf1 0x1071bc 0x106831 + +This will print the source code lines of the backtrace. + +To examine the backtrace of some given thread, use + + show all thread/u + +to get the whole listing of all tasks and threads. You can then use trace/t to trace a specific thread. + +Unfortunately, userland and kernelland use the same range of addresses, so one can not get userland traces easily. The Xen port uses different ranges, and in that case one can use trace/u to also get the userland trace. + +To examine a variable, use nm /boot/gnumach to get the address of the variable (e.g. 0x123400), and use + + x 0x123400 + +to read it. One can also write to it by using + + w 0x123400 + +Another interesting feature is watching a variable, by using + + watch 0x123400 + +and then type continue, to let Mach continue execution. The debugger will be entered again on any change in that variable. The watch is implemented in hardware, so it does not disturb or slow down execution at all. + + +# GDB in QEMU When you're [[running_a_system_in_QEMU|hurd/running/qemu]] you can directly [use GDB on the running kernel](http://www.nongnu.org/qemu/qemu-doc.html#SEC48). +# Code Inside the Kernel + Alternatively you can use an approach like this one: add the following code snippet to `device/ds_routines.c`'s `ds_device_open` function, right at the top of the function, and modify the code as needed. @@ -56,6 +115,8 @@ This is especially useful if you need to manually trigger some stuff inside the running kernel, as with the *D1* example. +## Writing to the Screen Buffer + If you're doing real low level debugging, you might want to put variations of the following snipped into the code, this code will write a `#` character at line `[LINE]`, column `[COLUMN]` on the screen: @@ -67,3 +128,22 @@ The call of `halt_cpu` will -- as the name suggests -- halt the system afterwards. This might be what you want or it might not, but it is needed at some place when running the kernel inside QEMU, as QEMU somehow decides not to update its display buffer anymore under certain conditions. + + +# Halting the CPU and Examining Registers + +IRC, freenode, #hurd, 2011-07-14: + + <braunr> one ugly trick i use when printf isn't available is to halt the + cpu + <braunr> then use info registers to know where the cpu is halted + <braunr> and you'll know if you reached that code or not + <braunr> (info registers is a qemu command) + + +# Serial Console + +IRC, freenode, #hurd, 2011-11-13: + + <youpi> use console=com0 + <youpi> to activate the console on the first serial port diff --git a/microkernel/mach/gnumach/hardware_compatibility_list.mdwn b/microkernel/mach/gnumach/hardware_compatibility_list.mdwn index 2152c079..874f5f07 100644 --- a/microkernel/mach/gnumach/hardware_compatibility_list.mdwn +++ b/microkernel/mach/gnumach/hardware_compatibility_list.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2009 Free Software Foundation, +[[!meta copyright="Copyright © 2007, 2008, 2009, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -6,8 +6,8 @@ id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] # CPU Architecture @@ -29,7 +29,7 @@ Read about further [[ports]]. # Memory -GNU Mach will use a maximum of 1 GiB of RAM. If your system has more, +GNU Mach will use a maximum of 1.7 GiB of RAM. If your system has more, the surplus will silently be ignored. (In past times, this would hinder GNU Mach from booting at all, but this has been fixed, so you no longer need to apply GRUB's `uppermem` directive.) @@ -68,10 +68,11 @@ All common IDE drives should work. Some drive geometries do not work, e.g. drives with hundreds of GiB of storage space, see [[!GNU_Savannah_bug 26425]]. -[[!toggle id="SATA" text="SATA drives may work in compatibility mode."]] -<!-- Sure? --[[tschwinge]] --> -[[!toggleable id="SATA" text=""" +## SATA + +SATA drives may work in compatibility mode. + This is how booting a [[GNU/Hurd_system|hurd]] will typically fail if GNU Mach couldn't connect to the hard disk, e.g., in a SATA system without IDE compatibility mode: @@ -81,7 +82,7 @@ compatibility mode: There *may* be an option in the system's BIOS setup to configure enabling such a compatibility mode. -"""]] + # Device Drivers diff --git a/microkernel/mach/gnumach/hardware_compatibility_list/discussion.mdwn b/microkernel/mach/gnumach/hardware_compatibility_list/discussion.mdwn index 69ca3190..2b65956a 100644 --- a/microkernel/mach/gnumach/hardware_compatibility_list/discussion.mdwn +++ b/microkernel/mach/gnumach/hardware_compatibility_list/discussion.mdwn @@ -1,4 +1,33 @@ +[[!meta copyright="Copyright © 2007, 2008, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + Further information may still be found on <http://www.nongnu.org/thug/gnumach_hardware.html> and could perhaps be incorporated into that page. --[[tschwinge]] + + +# SATA + +IRC, freenode, +hurd, 2011-07-24 + + <braunr> youpi: concerning the ide compatibility problem, it seems some + bioses provide several modes + <braunr> youpi: "legacy ide" and "native ide" + <braunr> i don't know what native ide really means, but when debugging ide + probing in gnumach, it just looks like there is nothing to detect + <braunr> and even in this mode, linux uses the ahci driver + <youpi> apparently native means it still uses the IDE protocol, but + possibly with other IRQs + <youpi> i.e. you need a PCI driver to handle that + <braunr> ok diff --git a/microkernel/mach/gnumach/memory_management.mdwn b/microkernel/mach/gnumach/memory_management.mdwn new file mode 100644 index 00000000..c630af05 --- /dev/null +++ b/microkernel/mach/gnumach/memory_management.mdwn @@ -0,0 +1,121 @@ +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation open_issue_gnumach]] + +[[!toc]] + + +# IRC, freenode, #hurd, 2011-02-15 + + <braunr> etenil: originally, mach had its own virtual space (the kernel + space) + <braunr> etenil: in order to use linux 2.0 drivers, it now directly maps + physical memory, as linux does + <braunr> etenil: but there is nothing similar to kmap() or vmalloc() in + mach, so the kernel is limited to its 1 GiB + <braunr> (3 GiB userspace / 1 GiB kernelspace) + <braunr> that's the short version, there is a vmalloc() in mach, but this + trick made it behave almost like a kmalloc() + <antrik> braunr: the direct mapping is *only* for the benefit of Linux + drivers?... + <braunr> also, the configuration of segments limits the kernel space + <braunr> antrik: i'm not sure, as i said, this is the short version + <braunr> antrik: but there is a paper which describes the integration of + those drivers in mach + <etenil> you mean the linux 2.0 drivers? + <antrik> braunr: I read it once, but I don't remember anything about the + physical mapping in there... + <antrik> etenil: well, originally it was 1.3, but essentially that's the + same... + <braunr> i don't see any other reason why there would be a direct mapping + <braunr> except for performance (because you can use larger - even very + lage - pages without resetting the mmu often thanks to global pages, but + that didn't exist at the time) + + +# IRC, freenode, #hurd, 2011-02-15 + + <antrik> however, the kernel won't work in 64 bit mode without some changes + to physical memory management + <braunr> and mmu management + <braunr> (but maybe that's what you meant by physical memory) + +## IRC, freenode, #hurd, 2011-02-16 + + <braunr> antrik: youpi added it for xen, yes + <braunr> antrik: but you're right, since mach uses a direct mapped kernel + space, the true problem is the lack of linux-like highmem support + <braunr> which isn't required if the kernel space is really virtual + + +# IRC, freenode, #hurd, 2011-06-09 + + <braunr> btw, how can gnumach use 1 GiB of RAM ? did you lower the + user/kernel boundary address ? + <youpi> I did + <braunr> 2G ? + <youpi> yes + <braunr> ok + <youpi> it doesn't make so much sense to let processes have 3G addressing + space when there can't be more that 1G physical memory + <braunr> that's sad for an operating system which does most things by + mapping memory eh + <youpi> well, if a process wants to map crazy things, 3G may be tight + already + <youpi> e.g. ext2fs + <braunr> yes + <youpi> so there's little point in supporting them + <braunr> we need hurd/amd64 + <youpi> and there's quite some benefit in shrinking them to 2G + <youpi> yes + <youpi> actually even 2G may become a bit tight + <youpi> webkit linking needs about 1.5-2GiB + <youpi> things become really crazy + <braunr> wow + <braunr> i remember the linux support for 4G/4G split when there was enough + RAM to fill the kernel space with struct page entries + + +# IRC, freenode, #hurd, 2011-11-12 + + <youpi> well, the Hurd doesn't "artificially" limits itself to 1.5GiB + memory + <youpi> i386 has only 4GiB addressing space + <youpi> we currently chose 2GiB for the kernel and 2GiB for the userspace + <youpi> since kernel needs some mappings, that leaves only 1.5GiB usable + physical memory + <sea4ever`> Hm? 2GiB for kernel, 2GiB for userspace, 500MiB are used for + what? + <youpi> for mappings + <youpi> such as device iomap + <youpi> contiguous buffer allocation + <youpi> and such things + <sea4ever`> Ah, ok. You map things in kernel space into user space then. + <youpi> linux does the same without the "bigmem" support + <youpi> no, just in kernel space + <youpi> kernel space is what determines how much physical memory you can + address + <youpi> unless using the linux-said-awful "bigmem" support + + +# IRC, freenode, #hurd, 2012-07-05 + + <braunr> hm i got an address space exhaustion while building eglibc :/ + <braunr> we really need the 3/1 split back with a 64-bits kernel + <pinotree> 3/1? + <braunr> 3 GiB userspace, 1 GiB kernel + <pinotree> ah + <braunr> the debian gnumach package is patched to use a 2/2 split + <braunr> and 2 GiB is really small for some needs + <braunr> on the bright side, the machine didn't crash + <braunr> there is issue with watch ./slabinfo which turned in a infinite + loop, but it didn't affect the stability of the system + <braunr> actually with a 64-bits kernel, we could use a 4/x split diff --git a/microkernel/mach/gnumach/ports.mdwn b/microkernel/mach/gnumach/ports.mdwn index a29b8651..f114460c 100644 --- a/microkernel/mach/gnumach/ports.mdwn +++ b/microkernel/mach/gnumach/ports.mdwn @@ -1,15 +1,24 @@ -[[!meta copyright="Copyright © 2007, 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2007, 2008, 2009, 2011 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] * x86. This is the main port. + + * [[Xen]] + * [PowerPC](http://www.pjbruin.dds.nl/hurd/). Is not in a usable state. - * Alpha. Was once started, but isn't in a usable state either. - * [[Xen]] + * Alpha: [project I](http://savannah.nongnu.org/projects/hurd-alpha), and + [project II](http://savannah.nongnu.org/projects/gnumach-alpha). Was once + started, but isn't in a usable state either. + + * MIPS. Status completely unknown. + + * [[open_issues/Mach_on_Top_of_POSIX]]. Status unknown. diff --git a/microkernel/mach/gnumach/ports/xen.mdwn b/microkernel/mach/gnumach/ports/xen.mdwn index c544fd57..c6023786 100644 --- a/microkernel/mach/gnumach/ports/xen.mdwn +++ b/microkernel/mach/gnumach/ports/xen.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2009 Free Software Foundation, +[[!meta copyright="Copyright © 2007, 2008, 2009, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -6,12 +6,13 @@ id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!toc]] -## Xen dom0, hypervisor + +# Xen dom0, hypervisor /!\ Now that GNU Mach handles PAE you can use a PAE-enabled hypervisor. @@ -20,7 +21,7 @@ You can either get binaries at <http://youpibouh.thefreecat.org/hurd-xen/> or bu - Copy `gnumach-xen-pae` and `hurd-modules` to your dom0 /boot. If you still have a non-PAE hypervisor, use `gnumach-xen-nonpae` instead. - Copy `hurd` into `/etc/xen`, edit it for fixing access to your hurd / and swap -## GNU/Hurd system +# GNU/Hurd system /!\ You need an already installed [[GNU/Hurd_system|hurd/running]]. @@ -30,7 +31,8 @@ If you have a free partition, you can fdisk to type 0x83, create a filesystem us Replace /dev/sda4 with your partition. Install and use crosshurd to setup a GNU/Hurd system on this partition. -## /etc/xen/hurd configuration + +# /etc/xen/hurd configuration Here is a sample /etc/xen/hurd configuration @@ -49,7 +51,8 @@ Suggestions about [[networking_configuration]] are available. If you need stable MAC addresses, use a syntax like `vif = [ 'mac=00:16:3e:XX:XX:XX, bridge=br0' ]`. -## Running Hurd with Xen + +# Running Hurd with Xen To run Hurd with Xen, use: @@ -63,7 +66,8 @@ and gnumach should get started. Proceed with native-install. - If `xm` complains about networking (`vif could not be connected`), it's Xen scripts' fault, see Xen documentation for how to configure the network. The simplest way is network-bridge with fixed IPs (note that you need the bridge-utils package for this). You can also just disable networking by commenting the vif line in the config. - If `xm` complains `Error: (2, 'Invalid kernel', 'xc_dom_compat_check: guest type xen-3.0-x86_32 not supported by xen kernel, sorry\n')`, you most probably have a PAE-enabled hypervisor and a non-PAE gnumach. Either install and boot non-PAE hypervisor and kernel, or rebuilt gnumach in PAE mode. -## Building from sources + +# Building from sources If you want to generate these images, first get the `gnumach-1-branch-Xen-branch` branch from gnumach CVS. Then look for "Ugly" in `kern/bootstrap.c`, how to generate `hurd-modules` is explained there, and you'll have to fix `EXT2FS_SIZE` and `LD_SO_SIZE` by hand. @@ -75,8 +79,49 @@ Then use The current `hurd-modules` was built from the debian packages `hurd 20070606-2` and `libc0.3 2.6.1-1`. /!\ This means that when using this image, your GNU/Hurd system also needs to be a glibc version 2.6 or later-based one! ---- +# `pv-grub` + +From Xen 4.0 on you can run the GNU Hurd directly using `pv-grub`, +without the need to [prepare a special bootstrap +image](http://youpibouh.thefreecat.org/hurd-xen/build_hurd-modules) (like an +initrd). + +Download http://youpibouh.thefreecat.org/hurd-xen/pv-grub.gz into /boot, and use the following for instance: + + kernel = "/boot/pv-grub.gz" + memory = 256 + disk = ['phy:sda4,hda,w'] + extra = "(hd0,1)/boot/grub/menu.lst" + vif = [ '' ] + +extra is now the path to the grub config file. + +# Partitions + +You will need the following notation for the gnumach root= parameter: + +root=part:2:device:hd0 + +to access the second partition of hd0, for instance. + +You will also need to use the parted storeio module for the /dev entries, for instance: + +settrans -fgap /dev/hd0s1 /hurd/storeio -T typed part:1:device:hd0 + +# Miscellaneous [[Internals]]. [[!GNU_Savannah_task 5468]], [[!GNU_Savannah_task 6584]]. + + +# Host-side Writeback Caching + +Optimization possible as it is with +[[QEMU|hurd/running/qemu/writeback_caching]]? + +IRC, freenode, #hurd, 2011-06-08 + + <braunr> youpi: does xen provide disk caching options ? + <youpi> through a blktap, probably + <braunr> ok diff --git a/microkernel/mach/gnumach/ports/xen/discussion.mdwn b/microkernel/mach/gnumach/ports/xen/discussion.mdwn new file mode 100644 index 00000000..2980e3b2 --- /dev/null +++ b/microkernel/mach/gnumach/ports/xen/discussion.mdwn @@ -0,0 +1,14 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + +Stuff from <http://youpibouh.thefreecat.org/hurd-xen> should be merged into +these pages here. diff --git a/microkernel/mach/gnumach/projects.mdwn b/microkernel/mach/gnumach/projects.mdwn index 47a2756c..f4ef192a 100644 --- a/microkernel/mach/gnumach/projects.mdwn +++ b/microkernel/mach/gnumach/projects.mdwn @@ -1,13 +1,13 @@ -[[!meta copyright="Copyright © 2005, 2006, 2007, 2008 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2005, 2006, 2007, 2008, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] This page is a place to keep track of ideas about things that may be improved in GNU Mach, so that it'll evolve to a reliable microkernel for The Hurd, both @@ -58,7 +58,8 @@ so that no duplicate efforts end up. * Improve the external pagers interface - * Implement read-ahead (huge I/O improvements expected). + * Implement [[open_issues/performance/io_system/read-ahead]] (huge I/O + improvements expected). * Making this interface synchronous should improve I/O performance significantly, without (almost) any drawbacks (we also get some diff --git a/microkernel/mach/gnumach/projects/clean_up_the_code.mdwn b/microkernel/mach/gnumach/projects/clean_up_the_code.mdwn index e865e61a..2a9b4b60 100644 --- a/microkernel/mach/gnumach/projects/clean_up_the_code.mdwn +++ b/microkernel/mach/gnumach/projects/clean_up_the_code.mdwn @@ -1,5 +1,5 @@ -[[!meta copyright="Copyright © 2005, 2006, 2007, 2008 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2005, 2006, 2007, 2008, 2010 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -9,6 +9,8 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] +[[!tag open_issue_gnumach]] + # Restructure the tree in a sane way Merge `linux/src` and `linux/dev`. But only if using a sane RCS, so leave it diff --git a/microkernel/mach/gnumach/projects/gdb_stubs.mdwn b/microkernel/mach/gnumach/projects/gdb_stubs.mdwn index ef1b4909..064da7bf 100644 --- a/microkernel/mach/gnumach/projects/gdb_stubs.mdwn +++ b/microkernel/mach/gnumach/projects/gdb_stubs.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2010 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -8,6 +8,12 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] +[[!tag open_issue_gnumach open_issue_gdb]] + * <http://lists.gnu.org/archive/html/bug-hurd/2008-04/msg00103.html> * [ChangeLog.gdb](http://cvs.savannah.gnu.org/viewvc/gnumach/ChangeLog.gdb?root=hurd&view=markup&pathrev=gnumach-1-branch-gdb-branch) + +This may be another follow-up project: [*Linux Kernel GDB tracepoint +module*](http://thread.gmane.org/gmane.comp.gdb.devel/29369), Hui Zhu, +2010-10-09. diff --git a/microkernel/mach/history.mdwn b/microkernel/mach/history.mdwn index a8951737..5a3608cd 100644 --- a/microkernel/mach/history.mdwn +++ b/microkernel/mach/history.mdwn @@ -1,7 +1,3 @@ -# <a name="Table_of_Contents"> Table of Contents </a> - -%TOC% - # <a name="Early_beginnings"> Early beginnings </a> Mach has quite a history. Everything actually started at the University of Rochester in 1975. It was invented to demonstrate how operating systems could be built using a modular design where processes communicated using message passing, even across networks. The system was called the Rochester Intelligent Gateway and ran on a 16 bit mini computer called Eclipse from Data General. diff --git a/microkernel/mach/ipc.mdwn b/microkernel/mach/ipc.mdwn index aaf3ba23..1bb44b59 100644 --- a/microkernel/mach/ipc.mdwn +++ b/microkernel/mach/ipc.mdwn @@ -1,22 +1,21 @@ -[[!meta copyright="Copyright © 2007, 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2007, 2008, 2010 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -[[General_information|/ipc]] about IPC. +Read about the [[general concept of *inter-process communication* (IPC)|/ipc]]. -An IPC is sent by invoking a [[port]]. <!-- Isn't this wording a bit strange? -``IPC is sent'' --tschwinge --> +On Mach, an IPC is done by invoking a [[port]]. + +The two fundamental operations, to *send* and *receive* [[message]]s, are used +to implement a [[RPC]] system. [[Sequence_numbering]]. [The Unofficial GNU Mach IPC beginner's guide](http://www.nongnu.org/hurdextras/ipc_guide/ipc_guide.html) - -# See Also - -* [[RPC]] diff --git a/microkernel/mach/memory_object.mdwn b/microkernel/mach/memory_object.mdwn new file mode 100644 index 00000000..f32fe778 --- /dev/null +++ b/microkernel/mach/memory_object.mdwn @@ -0,0 +1,33 @@ +[[!meta copyright="Copyright © 2002, 2003, 2010, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +Mach's [[virtual_memory]] subsystem uses *memory objects* for supplying the +content of regions of virtual memory in an [[virtual_address_space]]. + +All of these objects are managed by *memory manager*s, that are also called +*pager*s. These can be implemented as user-space processes. + +Both the memory objects, and their managers are kernel objects, and are +accessed by [[port]]s. + +A system's physical memory is conceived as a *memory cache* that contains +*memory cache objects*. So when a [[thread]] accesses a page in its task's +address space, the memory object that includes this page is *cached* in the +memory cache. Memory objects are [[paged out and paged +in|external_pager_mechanism]] by the aforementioned memory managers. The +decision when they should be paged in or paged out is left to [[Mach]]. Each +memory object has an ordered list of memory managers that provide paging. The +last one tried is the *default memory manager* that resides in the microkernel, +in contrast to most of the others. The default memory manager is needed +because the microkernel can't wait infinitely for someone else to free the +memory cache: it just calls the next memory manager hoping it to succeed. + +Read about [[GNU Mach's memory management|gnumach/memory_management]]. diff --git a/microkernel/mach/memory_object/discussion.mdwn b/microkernel/mach/memory_object/discussion.mdwn new file mode 100644 index 00000000..907f859a --- /dev/null +++ b/microkernel/mach/memory_object/discussion.mdwn @@ -0,0 +1,74 @@ +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation open_issue_gnumach]] + +[[!toc]] + + +# IRC, freenode, #hurd, 2011-08-05 + + < neal> braunr: For instance, memory objects are great as they allow you to + specify the mapping policy in user space. + < neal> braunr: But, the policy for determining the eviction order is + realized by Mach + < neal> braunr: And user-space has no control + < braunr> are you referring to the page cache lru approximation and stuff + like resource containers ? + < neal> I'm not sure what you mean by page cache lru appoximateion + < braunr> the kernel eviction policy :) + < neal> that's an implementation detail + + +# IRC, freenode, #hurd, 2011-09-05 + + <braunr> mach isn't a true modern microkernel, it handles a lot of + resources, such as high level virtual memory and cpu time + <braunr> for example, the page replacement mechanism can't be implemented + outside the kernel + <braunr> yet, it provides nothing to userspace server to easily allocate + resources on behalf of clients + <braunr> so, when a thread calls an RPC, the cpu time used to run that RPC + is accounted on the server task + <braunr> the hurd uses lots of external memory managers + +[[external_pager_mechanism]]. + + <braunr> but they can't decide how to interact with the page cache + <braunr> the kernel handles the page cache, and initiates the requests to + the pagers + <cjuner> braunr, why can't they decide that? + <braunr> because it's implemented in the kernel + <braunr> and there is nothing provided by mach to do that some other way + <slpz_> braunr: you probably already know this, but the problem with client + requests being accounted on behalf the server, is fixed in Mach with + Migrating Threads + +[[open_issues/mach_migrating_threads]]. + + <braunr> slpz_: migrating threads only fix the issue for the resources + managed by mach, not the external servers + <braunr> slpz_: but it's a (imo necessary) step to completely solve the + issue + <braunr> in addition to being a great feature for performance (lighter + context switchers, less state to track) + <braunr> it also helps priority inversion problems + <slpz_> braunr: I was referring just to cpu-time, but I agree with you an + interface change is needed for external pagers + <braunr> slpz_: servers in general, not necessarily pagers + <slpz_> as a way to mitigate the effect of Mach paging out to external + pagers, the folks at OSF implemented an "advisory pageout", so servers + are "warned" that they should start paging out, and can decide which + pages are going to be flushed by themselves + +[[open_issues/resource_management_problems]]. + + +# [[open_issues/memory_object_model_vs_block-level_cache]] diff --git a/microkernel/mach/message.mdwn b/microkernel/mach/message.mdwn new file mode 100644 index 00000000..ba47671e --- /dev/null +++ b/microkernel/mach/message.mdwn @@ -0,0 +1,31 @@ +[[!meta copyright="Copyright © 2002, 2003, 2010 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +*Messages* are collections of typed data, with a defined layout. + +They are used for [[IPC]], and are sent to and received from [[port]]s. + +These messages are not only opaque data. They can also contain [[port +rights|port]] to be passed to another [[task]]. Port rights are either +*copied* or *moved*. Notice that port receive right must be moved but not +copied because there can't be more than one task that holds the receive right +to a port. The receiving task creates new local port name to the port rights +it received. + +Some data in the message can be *out-of-line data*. In the message, these are +*references* to memory regions ([[memory_object]]s) that are *virtually +copied*. When the message is received in a task, these virtual copies become +part of the task by mapping them into the receiver's [[virtual_address_space]]. +Another key concept that is applied is using *copy-on-write*, which means that +data is not copied immediately, but only when it is changed. This is primarily +used to send large blocks of data efficiently, as it is too expensive to store +them in the kernel address space: extra copied need only be made at the moment +that the memory regions begin to diverge, by threads modifying them. diff --git a/microkernel/mach/mig.mdwn b/microkernel/mach/mig.mdwn index 4275a4b4..331b3bf4 100644 --- a/microkernel/mach/mig.mdwn +++ b/microkernel/mach/mig.mdwn @@ -1,21 +1,34 @@ -[[!meta copyright="Copyright © 2001, 2002, 2003, 2006, 2007, 2008 Free Software -Foundation, Inc."]] +[[!meta copyright="Copyright © 2001, 2002, 2003, 2006, 2007, 2008, 2010 Free +Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -The Mach Interface Generator (MIG) is an [[IDL]] compiler. Based on an -interface definition, it creates stubs to [[invoke]] object methods -and to demultiplex incoming messages. These stubs conveniently hide -the details of Mach's [[IPC]] machinery and make it easy to implement -and use Mach [[interface]]s as [[remote_procedure_calls_(RPC)|rpc]]. +The *Mach Interface Generator* (*MIG*) is an [[IDL]] compiler. Based on an +interface definition, it creates stub code to [[invoke]] object methods and to +demultiplex incoming messages. These stub functions conveniently hide the +details of Mach's [[IPC]] and [[port]] machinery and make it easy to implement +and use Mach [[interface]]s as [[remote procedure calls (RPC)|rpc]]: by using +the stub functions, the client programs can call remote procedures more or less +like any other C function. + +These functions encode arguments into [[message]]s' format (*marshalling*), +wait for a result on a newly created [[reply port|port]], decode return +arguments from the reply message (*demarshalling*, or *unmarshalling*) and pass +them to the client program. Similar actions are provided in the skeletons that +are linked to server programs. + +MIG allows very precise semantics to be specified about what the arguments are +and how to be passed. + + + * [[Documentation]] -* [[Documentation]] # Implementations diff --git a/microkernel/mach/mig/documentation.mdwn b/microkernel/mach/mig/documentation.mdwn index 82d51a72..7d4f1eca 100644 --- a/microkernel/mach/mig/documentation.mdwn +++ b/microkernel/mach/mig/documentation.mdwn @@ -1,13 +1,13 @@ -[[!meta copyright="Copyright © 2002, 2003, 2005, 2007, 2008, 2009 Free Software -Foundation, Inc."]] +[[!meta copyright="Copyright © 2002, 2003, 2005, 2007, 2008, 2009, 2010 Free +Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] This is a small collection of links to external documents describing the *Mach Interface Generator* used by GNU Mach. @@ -17,23 +17,23 @@ Interface Generator* used by GNU Mach. A tutorial which demonstrates the use of the C Threads library primitives in writing a multithreaded program and the use of the Mach Interface Generator -(MIG) to generate remote procedure calls for interprocess communication. Like +(MIG) to generate remote procedure calls for inter-process communication. Like its companion tutorial, it is based on the Mach 2.5 system. However, the concepts are applicable to Mach 3.0 user level programming. Linda R. Walmer and Mary R. Thompson. *A Programmer's Guide to the Mach User Environment*. [PostScript -](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/machuse.ps), -[Doc](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/machuse.doc). +](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/machuse.ps), +[Doc](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/machuse.doc). February 1988. School of Computer Science, Carnegie Mellon University. An ftp directory containing the [mig programming -examples](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig_example) +examples](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig_example) for this tutorial. Slides to Rich Drave's talk on MIG, on November 21, 1991: -[PostScript](ftp://ftp.cs.cmu.edu//afs/cs/project/mach/public/doc/unpublished/internals_slides/Mig/root.ps), -[TeX](ftp://ftp.cs.cmu.edu//afs/cs/project/mach/public/doc/unpublished/internals_slides/Mig/slides.tex). +[PostScript](http://www.cs.cmu.edu//afs/cs/project/mach/public/doc/unpublished/internals_slides/Mig/root.ps), +[TeX](http://www.cs.cmu.edu//afs/cs/project/mach/public/doc/unpublished/internals_slides/Mig/slides.tex). # Roots @@ -41,15 +41,15 @@ Slides to Rich Drave's talk on MIG, on November 21, 1991: Mig is an implementation of a subset of the Matchmaker **language**. "Matchmaker is a language for specifying and automating the generation of -multilingual interprocess communication interfaces. MIG is an interim +multilingual inter-process communication interfaces. MIG is an interim implementation of a subset of the Matchmaker language that generates C and C++ -remote procedure call interfaces for interprocess communication between Mach +remote procedure call interfaces for inter-process communication between Mach tasks." Richard P. Draves, Michael B. Jones, Mary R. Thompson, *MIG - THE MACH INTERFACE GENERATOR*. -[ps](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig.ps), -[doc](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig.doc). +[ps](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig.ps), +[doc](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig.doc). November 1989. Department of Computer Science, Carnegie-Mellon University. @@ -70,6 +70,12 @@ pp. 67--77." [Chapter 4, Inter Process Communication](http://www.gnu.org/software/hurd/gnumach-doc/Inter-Process-Communication.html). + * OSF's [Server Writer's Guide (ps)](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/server_writer.ps) + [Server Writer's Guide (pdf)](http://shakthimaan.com/downloads/hurd/server_writer.pdf) + + * OSF's [Server Writer's Interfaces (ps)](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/server_interface.ps) + [Server Writer's Interfaces (pdf)](http://shakthimaan.com/downloads/hurd/server_interface.pdf) + * Flags: * [[dealloc_and_dealloc[]|dealloc]] diff --git a/microkernel/mach/mig/gnu_mig.mdwn b/microkernel/mach/mig/gnu_mig.mdwn index 1bcbd545..0de1bd67 100644 --- a/microkernel/mach/mig/gnu_mig.mdwn +++ b/microkernel/mach/mig/gnu_mig.mdwn @@ -1,13 +1,13 @@ -[[!meta copyright="Copyright © 2001, 2006, 2008, 2009 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2001, 2006, 2008, 2009, 2010 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] GNU MIG is the GNU distribution of the [[Mach_3.0_interface_generator_*MIG*|mig]], as maintained by the GNU Hurd @@ -20,5 +20,9 @@ software in the GNU system that uses Mach-based GNU MIG is fully compatible with [[OSF_MIG|mig]]. +Like its predecessor, it can only generate C code, that has to be compiled and +linked to client and server programs respectively ([[!taglink +open_issue_mig]]). + * [[Building]] - building (and obtaining) GNU MIG * [[Open Issues|tag/open_issue_mig]] diff --git a/microkernel/mach/mig/gnu_mig/building.mdwn b/microkernel/mach/mig/gnu_mig/building.mdwn index f92f7dbe..e7d3c150 100644 --- a/microkernel/mach/mig/gnu_mig/building.mdwn +++ b/microkernel/mach/mig/gnu_mig/building.mdwn @@ -1,15 +1,28 @@ +[[!meta copyright="Copyright © 2006, 2007, 2008, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + # <a name="Building_the_Mach_Interface_Gene"> Building the Mach Interface Generator from Source </a> -If you want to build the Mach Interface Generator yourself instead of just using a pre-built package, follow these instructions. +If you want to build the Mach Interface Generator yourself instead of just +using a pre-built package, follow these instructions. ## <a name="Getting_the_Source_Code"> Getting the Source Code </a> You can chose between getting the [sources from the developers' -RCS](http://savannah.gnu.org/cvs/?group=hurd): +RCS](http://git.savannah.gnu.org/cgit/hurd/): - $ cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/sources/hurd co mig + $ git clone http://git.savannah.gnu.org/cgit/hurd/mig.git/ -... or (if you are working on a Debian system) the ones that are used for the [current Debian mig package](http://packages.debian.net/source/unstable/mig): +... or (if you are working on a Debian system) get the sources that are used for the +[current Debian mig package](http://packages.debian.net/source/unstable/mig): $ apt-get source mig @@ -17,53 +30,70 @@ Please see the Debian [[hurd/running/debian/FAQ]] before using _apt-get source_. The unpacked source tree is around 1 MiB, and the build tree also is around 1 MiB. -## <a name="Preparing_for_the_Build"> Preparing for the Build </a> +## <a name="_on_Debian_systems"> On Debian Systems: </a> -### <a name="_on_Debian_systems"> ... on Debian systems </a> +### <a name="Preparing_for_the_Build"> Preparing for the Build </a> -Building the Mach Interface Generator requires the _build-essential_ and _fakeroot_ packages, their dependencies and additional packages that are specified by the source mig package: +Building MIG requires the *build-essential* and *fakeroot* packages, +and some additional dependencies specified by the mig source package: # apt-get install build-essential fakeroot # apt-get build-dep mig -### <a name="_on_non_Debian_systems"> ... on non-Debian systems </a> +### <a name="Building_and_Installing"> Building and Installing </a> <a name="_a_deb_file"> ... a _.deb_ file </a> -Building the Mach Interface Generator requires a C compiler, a standard C library (with corresponding header files) and your favourite flavor of awk (gawk), yacc (bison), lex (flex) and make. +Change into the directory with the downloaded / unpacked MIG sources: -Additionally, you need to have GNU Mach's header files installed. See -[[mach/gnumach/building]] about how to do that, then come back here. + $ cd mig-X.X.X.XX -## <a name="Building_and_Installing"> Building and Installing </a> +Start the build process: -### <a name="_a_deb_file"> ... a _.deb_ file </a> + $ dpkg-buildpackage -us -uc -b -rfakeroot -Change into the directory with the downloaded / unpacked MIG sources (_mig-1.3.1.99_): +This will create a _.deb_ package in the parent directory, +which you can then install on your system. - $ cd mig-1.3.1.99 +## <a name="_on_non_Debian_systems"> On non-Debian Systems: </a> -Start the build process: +### <a name="Preparing_for_the_Build"> Preparing for the Build </a> - $ dpkg-buildpackage -us -uc -b -rfakeroot +Building the Mach Interface Generator requires a C compiler, a standard 32 bit +C library (with corresponding header files), your favourite flavor of awk +(gawk), yacc (bison), lex (flex) and make. -You can then install / distribute the _.deb_ file which will drop out one directory above the current one. +Additionally, you need to have GNU Mach's header files installed. See +[[building GNU Mach|mach/gnumach/building]] about how to do that, then come back here. + +### <a name="Building_and_Installing"> Building and Installing </a> + +First, generate the configuration files: -### <a name="_TODO_"> [TODO] </a> + $ cd mig + $ autoreconf --install -The Mach Interface Generator has to be built in a separate directory: +The Mach Interface Generator has to be built in a separate build directory: - $ mkdir mig-build - $ cd mig-build + $ mkdir build + $ cd build + +Find the base directory where you installed GNU Mach's header files and where +you now intend to install the Mach Interface Generator (e.g. _~/gnu_), and run +configure: + + $ GNU=~/gnu + $ TARGET_CPPFLAGS=-I"$GNU"/include ../configure --prefix="$GNU" -Find the root directory where you installed GNU Mach's header files and where you now intend to install the Mach Interface Generator (_~/gnu_) and the path to your Mach Interface Generator sources (\_[...]/mig) and configure it: +If you are building on a 64 bit machine, you need to add a --host option: $ GNU=~/gnu - $ TARGET_CPPFLAGS=-I"$GNU"/include [...]/mig/configure --prefix="$GNU" + $ TARGET_CPPFLAGS=-I"$GNU"/include ../configure --prefix="$GNU" --host=i686-unknown-linux-gnu -Build and install the Mach Interface Generator into _$GNU_, i.e. _~/gnu/_ in our example: +Build and install the Mach Interface Generator into _$GNU_ (i.e. _~/gnu/_ in our example): $ make all install -To make your _mig_ binary easily available, you should append something like the following to e.g. your _~/.bash\_profile_: +To make your _mig_ binary easily available, you should append something like +the following to e.g. your _~/.bash\_profile_: PATH=~/gnu/bin:$PATH export PATH diff --git a/microkernel/mach/mig/gnu_mig/building/discussion.mdwn b/microkernel/mach/mig/gnu_mig/building/discussion.mdwn new file mode 100644 index 00000000..d7636158 --- /dev/null +++ b/microkernel/mach/mig/gnu_mig/building/discussion.mdwn @@ -0,0 +1,16 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +# Non-cross-compiling + +[[!tag open_issue_mig]] + +[[samuelthibault]] mentioned that I should make clear what compiler options, etc. are only needed if compiling on a 64 bit computer. However, I don't know if the --host=i686... option is needed, here and when making gnumach, in case there may be some other default on 32 bit computers? --[[sudoman]] + diff --git a/microkernel/mach/pmap.mdwn b/microkernel/mach/pmap.mdwn new file mode 100644 index 00000000..6910bfd3 --- /dev/null +++ b/microkernel/mach/pmap.mdwn @@ -0,0 +1,74 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation open_issue_gnumach]] + + +# IRC, freenode, #hurd, 2012-02-01 + + <sekon> on Hurd what is the difference between kernel memory object and + pmap module ?? + <sekon> pmap is heap/libraries table for each thread while kernel memory + object refers to arbitary blobs of data ?? + <braunr> sekon: pmap is the low level memory mapping module + <braunr> i.e. it programs the mmu + <braunr> and these aren't hurd-specific, they are mach modules + <sekon> braunr: so kernel memonry objects consists of a bunch of pmaps ?? + <braunr> sekon: memory objects can be various things, be specific please + <braunr> (they're certainly not a bunch of pmaps though, no) + <braunr> there is one pmap per vm_map, and there is one vm_map per task + <braunr> and there is no need for double question marks, is ther ?? + <sekon> lol then is kernel memory object , please excuse the metaphor + something like a base class for pmap + <braunr> i don't know what a "kernel memory object" is, be specific please, + again + <sekon> braunr: + http://courses.cs.vt.edu/~cs5204/fall05-gback/presentations/MachOS_Rajesh.ppt + <sekon> goto page titled External Memory Management (EMM) on page 15 + <sekon> Kernel memory object shows up + <braunr> you know there are other formats for this document + <sekon> nope .. i did not know that + <sekon> in page 17 pmamp shows up + <braunr> "the problems of external memory management" ? + <sekon> braunr: the paper i am also reading is called x15mach_thesis + <braunr> ah, that's mine + * sekon bows + <sekon> :) + <braunr> ok i see page 17 + <sekon> so please good sir explain the relationship between kernel memory + object and pmap + <sekon> (if any) + <sekon> braunr: there is no mention of kernel memory object + <braunr> again, i don't see any reference or definition of "kernel memory + object" + <sekon> but your paper says + <sekon> that when page faults occur + <sekon> the kernel contact the manager for a kernel reference object + <sekon> *memory + <braunr> where ? + <sekon> in section 2.1.3 (unless i read it wrong) + <sekon> no just a sec + <sekon> 2.1.5 + <braunr> i never used the expression "kernel memory object" there :p + <braunr> anyway, you're referring simple to memory objects as seen by + userspace pagers + <braunr> a memory object is a data container + <braunr> usually, it's a file + <braunr> but it can be anything + <braunr> the pager is the task that provides its content and implements the + object methods + <braunr> as for the relation between them and the pmap module, it's a + distant one + <braunr> i'll explain it with an example + <braunr> page fault -> request content of memory object at a given offset + with given length from pager -> ask pmap to establish the mapping in the + mmu + <sekon> braunr: thank you ver much + <sekon> *very diff --git a/microkernel/mach/port.mdwn b/microkernel/mach/port.mdwn index af4a0c8d..26b55456 100644 --- a/microkernel/mach/port.mdwn +++ b/microkernel/mach/port.mdwn @@ -1,41 +1,89 @@ -[[!meta copyright="Copyright © 2007, 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2002, 2003, 2007, 2008, 2010, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] - -Mach ports are [[capabilities|capability]]. - -A Mach port is a kernel queue. Each port has associated with -it a receive right and one or more send and send-once rights. -A queue can hold a number of messages. Once the queue is full, -the send blocks until their is space to enqueue the message -(this is interruptible via a timeout mechanism). - -A receive right designates a queue and authorizes the holder to -dequeue messages from the queue, and to create send and send-once -rights. - -Send and send-once rights designate a queue and authorize the -hold to enqueue messages (in the case of a send-once right, -a single message). Enqueuing a message is equivalent to -[[invoke|invoking]] a capability. - -Send and receive rights are named using local names. Each -task has associated with it a port [[address_space]]. A ports -are addressed via this table. Each task thus has its own -private [[naming_context]] for ports. - -Ports can be [[delegate]]d in an [[IPC]] message. When the -receiver dequeues the message, the right is made available -to it. - -A [[thread]] can only block receiving on a single port. To work -around this, the concept of a port set was introduced. A receive -right can be added to (at most) one port set. When a thread -receives from a port set, it dequeues from any of the ports that -has a message available. +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[Mach]] *port*s are [[capabilities|capability]], and are also essentially +similar to [[UNIX]] pipes. They are unforgeable communication channels, +implemented by kernel queues. + +Each port has associated with it one *receive right* and one or more *send +right*s and *send-once right*s. That is, there is one receiver and one or more +senders -- a unidirectional communication channel. Only with the corresponding +port right, access to a port is possible; this is enforced by Mach. + +The kernel queue can hold a number of [[message]]s. Once the queue is full, +the send blocks until there is space to enqueue the message (this is +interruptible via a timeout mechanism). + +A receive right [[designates|designation]] a queue and authorizes the holder to +dequeue messages from the queue, and to create send and send-once rights. + +Send and send-once rights designate a queue and authorize the hold to enqueue +messages (in the case of a send-once right, a single message). Enqueuing a +message is equivalent to [[invoke|invoking]] a capability. + +Ports are automatically destroyed when there is no associated port right to +them. + +Mach knows what port rights belong to each task, but [[thread]]s that running +in the context of a task refer to ports by means of send and receive rights +that are named using local *port names*. These port names are plain integers, +like [[UNIX file descriptors|unix/file_descriptor]]. Only these local names +can be used by [[thread]]s for invoking operations on ports, threads do not +deal with port rights directly. + +For that, each task has associated with it a *port address space*, or *port +name space*. All ports are addressed via this table. Each task thus has its +own private [[naming_context]] for port rights. + +So, the picture is that after obtaining a port send right, the client uses a +port name to send [[message]]s to the port, or exactly one message if it's a +send-once right. These messages are (probably) queued and when the server task +tries to receive messages by having a [[thread]] use its port receive right, it +gets the message(s). This is called [[IPC]]. + +Port rights themselves can be [[delegate]]d in a [[message]], too. When the +receiver dequeues the message, the right is made available to it. + +The delivery of [[message]]s is reliable and strictly ordered. When a +[[thread]] sends messages *1* and *2*, it is guaranteed that the receiving +[[task]] will catch them in the same order. Of course, there can be +intermediate messages that are sent by other threads. + +Ports are objects that are implemented by the [[kernel]], and they are +kernel-protected resources: they are unforgeable, and there is no way for a +[[task]] to do anything with a port unless it have corresponding port right. + +Due to this, ports are globally unique. This makes them ideal for constituting +system-wide *object references*. (Fruther reading: +{{$capability#wikipedia_object-capability_model}}.) For example, the [[RPC]] +system as used by the GNU Hurd works by invoking *methods* on such object +references. The available methods are defined in [[hurd/interface]] files, and +are processes by the [[MIG]] tool. + +Invoking an operation on a port does not transfer the current execution control +to the receiver, but instead is an asynchronous operation. For this, and +especially in a [[RPC]] system, the sender may include a *reply port* using a +send-once right, and synchronize (block) on that one. + + +# Port Set + +A [[thread]] can only block receiving on a single port. To work around this, +the concept of a *port set* was introduced. A receive right can be added to +(at most) one port set. These port sets look like port receive rights, but +cannot be passed to other tasks, and there are additional operations for adding +and removing port receive rights. + +When a server process' thread receives from a port set, it dequeues exactly one +message from any of the ports that has a message available in its queue. + +This concept of port sets is also the facility that makes convenient +implementation of [[UNIX]]'s `select` [[system_call]] possible. diff --git a/microkernel/mach/rpc.mdwn b/microkernel/mach/rpc.mdwn index 72acfaa0..422e0441 100644 --- a/microkernel/mach/rpc.mdwn +++ b/microkernel/mach/rpc.mdwn @@ -1,15 +1,28 @@ -[[!meta copyright="Copyright © 2007, 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2002, 2003, 2007, 2008, 2010, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -[[General_information|/rpc]] about RPC. +Read about the [[general concept of a *remote procedure call* (RPC)|/rpc]]. Uses Mach's [[IPC]] [[mechanism]]. -Stub code generated by [[MIG]]. +The [[port]] abstraction allows RPCs to be executed on another computer +transparently. This can be implemented with user [[task]]s, but there is an +implementation in the kernel possible, too, which is called *NORMA*, but is not +avilable in [[GNU Mach|gnumach]]. + +The RPC stub code generated by [[MIG]]. + + +# Tracing + + * [[hurd/debugging/rpctrace]] + + * [[open_issues/librpci]] diff --git a/microkernel/mach/rpc/discussion.mdwn b/microkernel/mach/rpc/discussion.mdwn new file mode 100644 index 00000000..00e4a012 --- /dev/null +++ b/microkernel/mach/rpc/discussion.mdwn @@ -0,0 +1,117 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + + +# IRC, freenode, #hurd, 2011-06-11 + + <antrik> I don't think we have a precendence case of Mach initiating RPCs + to userspace tasks + <braunr> well mach regularly sends RPCs to external pagers + <antrik> hm, right + <antrik> anyways, the ds_ in device.defs is for use *inside* Mach, not for + the userspace interface + <braunr> what makes you think so ? + <antrik> several things + <antrik> not least the fact that without zhengda's modifications, the + device handling never calls out to userspace for all I know + <braunr> hm, it does + <braunr> for async I/O + <braunr> when the kernel has finished its I/O, it calls + ds_device_read_reply/ds_device_write_reply + <antrik> I see + <antrik> I never quite understood the _reply stuff + <braunr> although i wonder how mig is supposed to forge those names + <antrik> braunr: it isn't + <antrik> braunr: there is a separate device_reply.defs + <antrik> braunr: and it sets a *userprefix* of ds_ + <antrik> rather than a serverprefix + <braunr> i saw, yes + <braunr> ah right + <antrik> so ds still refers to the in-Mach device server, not anything + userspace + <braunr> so this is where the patch is supposed to introduce the + device_intr_notify RPC + <antrik> or at least that's my understanding... + <braunr> no, it doesn't refer to in-mach servers + <braunr> it really forges the right rpcs to be called by mach + <antrik> the definition of "RPC" is rather unclear here + <braunr> why ? + <braunr> mach has its own mach_msg() call for kernel-to-user messaging + <antrik> yes, but this is used only to send the reply message for the RPC + earlier initiated by userspace AIUI + <antrik> it doesn't look like there is any special RPC for async I/O + <braunr> yes, because this is the only use case they had + <braunr> hence the name "reply" + <braunr> intr_notify isn't a reply, but it uses the same mechanism + <braunr> these are declared as simpleroutine + <antrik> sure. but the fact that it isn't a reply message, but rather + initiates a new RPC, changes things from MiG point of view I believe + <antrik> right, as there is no reply to the reply :-) + <braunr> :) + <braunr> a simpleroutine is how to turn an rpc into a simple ipc + <antrik> I know + <antrik> so in _reply, we pretend that the reply is actually a new RPC, + with server and client roles reversed, and no reply + <antrik> (this is actually rather kludgy... apparently MIG has no real + notion of async replies) + <braunr> i don't understand what you mean + <braunr> simpleroutine is the explicit solution for async replies + <braunr> as stated in + http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig.ps + <braunr> it's not a new rpc with roles reversed + <braunr> it's not a reply either + <antrik> it might be an explicit solution for that, but it still seems + kludgy :-) + <braunr> i don't see why :/ + <braunr> would you have expected something like an option to create both + sync and async versions ? + <antrik> because it requires an extra .defs file + <antrik> yes + <braunr> ok + <braunr> well this seems cumbersome to me :) + <braunr> i prefer the simpleroutine approach + <braunr> but i agree this seems odd since mach has a high level ipc api + <antrik> anyways, my point is that the ds_ in device_reply.defs still + refers to the Mach side of things + <braunr> npnth: which package fails to build ? + <antrik> though a userspace process that actually handles the replies in an + async fashion will of course need some kind of device server too, just + like the DDE stuff... + <antrik> though naming it ds_ is confusing IMHO, because of the name clash + with the device server in Mach + <braunr> hm again, i fail to see why + <braunr> ds_ just means device_server + <braunr> and as most things in mach, it can be in kernel or not + <braunr> i mean, this is an interface prefix, i don't refer to an actual + single instance of a "device server" out there + <antrik> oh, right... DDE implements the Mach device protocol, so it *does* + do the ds_ part... but that makes the interrupt notification stuff even + more confusing + <braunr> hm + <braunr> because it provides a ds_device_intr_notify() which will never be + used, just to completely implement the interface ? + <antrik> yeah, that's what I suspect... + <braunr> sounds likely + <antrik> the device interface actually has two parts: one for "generic" + RPCs on the master device port, and one for device-specific RPCs. DDE + implements the latter, and uses the former... + <antrik> they live in separate places though I think: the individual device + RPCs are implemented in libmachdev, while the intr_ stuff is used in + libddekit probably + <braunr> it would be hairy to build otherwise + <antrik> so we *really* need to know what component npnth gets the error + with + <antrik> braunr: nah, not really. that's why we always have a separate + prefix for the server routines in Hurd RPCs + <braunr> right, i really need to read about mig again + <antrik> it's pretty normal for a translator to both implement and use an + interface diff --git a/microkernel/mach/task.mdwn b/microkernel/mach/task.mdwn new file mode 100644 index 00000000..c03c6a14 --- /dev/null +++ b/microkernel/mach/task.mdwn @@ -0,0 +1,23 @@ +[[!meta copyright="Copyright © 2002, 2003, 2010 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +A Mach *task* is a collection of resources, a [[virtual_address_space]], and a +[[port name space|port]]. They depend on [[thread]]s for executing program +code: a task alone has no means to do so. + +Switching from one task to another one involves doing a *context switch*, which +is usually not a cheap operation, as it involves switching the hardware's idea +of the memory layout ([[virtual_address_space]]), amongst others. + +Mach tasks are distinct from [[UNIX processes|unix/process]] in that they +provide less facilities. In processes, there are [[unix/signal]]s, process / +group / session IDs, [[unix/file_descriptor]]s and many other things. Tasks +are used for resource allocation and sharing; they are *resource container*s. diff --git a/microkernel/mach/thread.mdwn b/microkernel/mach/thread.mdwn new file mode 100644 index 00000000..e27bb117 --- /dev/null +++ b/microkernel/mach/thread.mdwn @@ -0,0 +1,37 @@ +[[!meta copyright="Copyright © 2002, 2003, 2010 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +A Mach *thread* belongs to exactly one [[task]], and is the means of execution. +The task supplies the resources. + +Mach threads are implemented inside the [[kernel]], as opposed to other +systems' user-level thread packages. + +A thread (theoretically) runs concurrently with all the other threads of a +system. If the system provides several processors, they can be used for +simultaneously running either several threads of the same task, or several +threads of different tasks. <!-- This is called SMP; the processors use +*shared memory*. --> [[!tag open_issue_documentation]] <!-- This needs a new +page, also covering Mach's `processor_set`s, and non-SMP, but still +multiprocessor systems. --> (But this is currently not support in [[GNU +Mach|gnumach]].) + +It is easy for the kernel to switch execution from one thread to another one +inside the same task: essentially, it only involves exchanging a few processor +registers' state. + +Threads have scheduling parameters and maintain various statistics about +themselves. + +On GNU/Hurd, APIs for Mach threads and thereabouts are provided by the +[[hurd/libthreads]] (cthreads), and [[libpthread]] (POSIX Threads) packages. + +A task backing a thread is the basis for a [[UNIX process|unix/process]]. diff --git a/microkernel/mach/virtual_address_space.mdwn b/microkernel/mach/virtual_address_space.mdwn new file mode 100644 index 00000000..97bc5f6b --- /dev/null +++ b/microkernel/mach/virtual_address_space.mdwn @@ -0,0 +1,36 @@ +[[!meta copyright="Copyright © 2002, 2003, 2010 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +*Virtual address space*s in Mach define the valid virtual addresses that can be +used by [[thread]]s under execution in the [[task]] that owns that address +space. Each task has only one address space and each address space belongs to +only one task. So when we want to name an address space (for example, in the +Mach API) we name it by the task it belongs to. + +These address spaces are divided into *pages*. Each page has individual +properties like *access rights* (*read* / *write* / *execute*), *inheritance +attributes* (*no inheritance* / *copy* / *share*) and some other system +properties. Page manipulation is optimized to help moving large blocks of data +from one address space to another, for example when one thread provides data to +another thread -- *client / server* technology. + +Memory ranges of pages that can be controlled as a whole are called +*[[memory_object]]*s. + +*Wired pages* are those that cannot be [[paged out|external_pager_mechanism]]. +For example, Mach itself is a task with its own address space and threads, and +all of its pages are wired. + +*Precious pages* are those that must not be discarded silently when they are +clean and memory is needed. For example, a memory manager that shares memory +across a network could not restore a page if it is silently discarded because +it is unmodified. This is not valid for the well-known [[pager +managers|external_pager_mechanism]] that use disks as backing store. diff --git a/microkernel/viengoos.mdwn b/microkernel/viengoos.mdwn index d4edc929..66c6ff36 100644 --- a/microkernel/viengoos.mdwn +++ b/microkernel/viengoos.mdwn @@ -1,15 +1,26 @@ -[[!meta copyright="Copyright © 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -*viengoos* is a new kernel currently being designed and written by Neal -Walfield. +*Viengoos* is a research kernel, designed and written by Neal Walfield. + +As of late 2009, the project is on hold, due to time constraints. + +Viengoos is not really meant to be a successor to [[Mach]]. It is highly +experimental; some of the techniques it employs, in particular, those related +to [[memory_management]] and [[IPC]], are unproven. These were motivated by +[[shortcomings_in_Mach|hurd/critique]] as well as current operating systems. A +research system is unlikely the best base for a product. A better approach is +to view Viengoos as an experimental platform whose goal is to explore solutions +to some of the [[issues_uncovered_by_the_Hurd|challenges]]. Knowledge gained +can then be integrated into something like [[Mach]]. The source can be downloaded from the *viengoos.git* repository, cf. <http://git.savannah.gnu.org/gitweb/?p=hurd/viengoos.git>. You can @@ -24,6 +35,8 @@ Then update to viengoos-on-bare-metal viengoos-on-bare-metal is the current development focus. +Discussion should be held on the [[mailing lists/l4-hurd]] mailing list. + * [[Building]] * Running * [[QEMU]] diff --git a/microkernel/viengoos/documentation.mdwn b/microkernel/viengoos/documentation.mdwn index 52ff7a48..edcc79a7 100644 --- a/microkernel/viengoos/documentation.mdwn +++ b/microkernel/viengoos/documentation.mdwn @@ -1,12 +1,12 @@ -[[!meta copyright="Copyright © 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2012 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] The most up-to-date documentation is in the source code itself, see in particular the header files in the hurd directory. @@ -17,7 +17,8 @@ version of that is available [[here|reference-guide.pdf]]. It is not, however, automatically regenerated, and thus may not be up to date. -Academic Papers: + +# Academic Papers * [Viengoos: A Framework for Stakeholder-Directed Resource Allocation](http://walfield.org/papers/2009-walfield-viengoos-a-framework-for-stakeholder-directed-resource-allocation.pdf). @@ -54,3 +55,8 @@ Academic Papers: argue that only a small static number of scheduling policies are needed in practice and advocate hierarchical policy specification and central realization. + + +# Miscellaneous + + * [[IRC_2012-02-23]] diff --git a/microkernel/viengoos/documentation/irc_2012-02-23.mdwn b/microkernel/viengoos/documentation/irc_2012-02-23.mdwn new file mode 100644 index 00000000..a3229be9 --- /dev/null +++ b/microkernel/viengoos/documentation/irc_2012-02-23.mdwn @@ -0,0 +1,159 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="IRC, freenode, #hurd, 2012-02-23"]] + +[[!tag open_issue_documentation open_issue_viengoos]] + + <braunr> neal: i've read a bit about current modern microkernel based + systems, and i'm wondering + <braunr> neal: can a capability be used for both request and replies, or + does messaging need something similar to reply ports ? + <neal> braunr: you want a reply port + <neal> think about a file server: + <neal> the file server publishes a capability to access something + <neal> and multiple entities use it + <neal> if you wanted just bidirectional caps + <braunr> that's the idea i had in mind, i just wondered if it was actually + still the case in practice + <neal> you'd need to create a new capability every time you delegated the + cap + <braunr> yes + <braunr> thanks + <braunr> what about send once rights ? + <neal> also, if you send on a cap and then start waiting on it you could + get your own reply :) + <neal> you can get around send-once rights by using a counter + <braunr> no i mean, is their behaviour still needed/useful ? + <neal> the counter is kernel implemented + <neal> yes + <neal> as an optimization + <braunr> so they're just a special case of capability + <neal> yes + <braunr> not a special capability type of their own + <neal> but they eliminate the constant create/destroy sequence + <braunr> (even if it was already the case at the implementation level in + mach, they were named separately which could confuse people) + <braunr> hm + <braunr> actually, send once rights were used for important notifications + such as dead port notifications + <braunr> is this still handled at the kernel level in modern ukernels ? + <neal> in viengoos, this is called the version field + <neal> see chapter 2 + <neal> + http://www.gnu.org/software/hurd/microkernel/viengoos/documentation/reference-guide.pdf + <braunr> neal: btw, congratulations for viengoos, it really is a very + interesting project: ) + <neal> thanks + <braunr> i don't see the point of rewriting a mach clone after reading + about it eh + <neal> I would definately do the messenger concept again + <neal> but I'd not do persistence + <braunr> i don't fully understand how messengers deal with blocking + <neal> did you read chapter 4? + <braunr> i read all of it but didn't understand everything :) + <braunr> it's quite abstract and i didn't make time to read some of the + source code + <neal> If you have specific questions, I can try to help + <braunr> i'll read those chapter again and formulate my questions after + <neal> I may have to read them as well :) + <braunr> i don't understand how you manage to separate IPC from threading + actually + <braunr> are messengers queues ? + <neal> messengers are super-buffers + <neal> they contain a reference to a thread object + <neal> to send a message, I use a messenger + <neal> I put the data in a buffer + <neal> and then I attach the messenger to the target messenger + <antrik> braunr: my stance is that we should try to incorporate the ideas + from Viengoos into Mach in an evolutionary process... + <neal> this causes an activation to be sent to the target messenger's + thread object + <braunr> neal: which activation ? + <neal> an activation is like a CPU interrupt + <braunr> neal: is it "allocated" at that moment, or taken from the sending + thread ? + <braunr> (i'm not sure my question really makes sense to you :/) + <antrik> braunr: not sure what you are asking exactly; but the basic idea + is that the receiving process preallocates message buffers + <braunr> antrik: maybe, i'm not sure + <antrik> when someone sends a message, it's stored in one of these buffers, + and the process gets a scheduler activation, so it can decide what to do + with it + <neal> antrik is right + <neal> the traget messenger designates a memory buffer + <braunr> i'm wondering about the details of this activation + <braunr> is it similar to thread migration ? + <neal> just before the activation, the data is copied to the messenger's + buffer + <neal> now someone needs to be notified + <neal> (that a message arrived) + <neal> that someone is the thread designated in the target messenger's + thread field + <neal> this is done by an activation + <neal> an activation is just an upcall + <neal> a thread is forced to a particular IP + <neal> an activation isn't a "what" it's a "how" + <neal> I never understood thread migration + <neal> as it's not really about threads + <neal> nor it is about migration + <antrik> neal: what happens if another message comes in before the + activation handling tread is done with the previous one?... + <neal> the messenger is enqueued on the thread object + <neal> it is delivered when the thread is in normal mode + <neal> part of delivering an activation is putting the thread is activation + mode + <neal> when in activation mode, it can't receive any activations + <braunr> i see + <braunr> but then, when a thread receives an activation, does it handle + several queued messengers at once (not to loose events/messages) ? + <neal> (unless it does a blocking receive on a particular messenger, which + is necessary to support memory allocation in activated mode) + <neal> it handles one at a time + <braunr> ah right + <neal> it can't lose events + <braunr> activations are sent per messengers/events + <neal> well, it can + <neal> but it is possible to prevent this + <braunr> neal: also, is message passing completely atomic ? + <neal> I'm not sure what you mean + <neal> which part + <braunr> well, all parts of a message :) + <braunr> in mach, a message can contain several parts + <braunr> data, rights, passing one of them may fail + <braunr> only the header is atomically processed + <neal> it's not atomic in the sense that a thread can observe the data copy + <braunr> that's not what i meant + <braunr> is a message completely transferred or not at all in case of + failure ? + <neal> it may be partially transferred + <braunr> or can it be partially transferred + <braunr> ok + <neal> for instance, if the target thread doesn't provide a memory buffer + <neal> then the data can't be copied + <neal> I don't recall off hand how I dealt with bad addresses + <neal> may be it is not possible + <neal> I don't remember + <neal> sorry + <braunr> but if i read the message structure correctly, there can be one + data block, and several capability addresses in a single message, right ? + <neal> yes + <braunr> ok + <braunr> have you considered passing only one object (either data or + capability) per message ? + <braunr> or is it too inefficient ? + <neal> you at least need a reply port + <neal> s/port/messenger/ + <braunr> yes but can't it be passed separately ? + <neal> then you have server state + <neal> ik + <braunr> hm yes + <braunr> thanks for your answers: ) + <neal> no problem diff --git a/microkernel/viengoos/projects.mdwn b/microkernel/viengoos/projects.mdwn index 27dcc3e2..971206bb 100644 --- a/microkernel/viengoos/projects.mdwn +++ b/microkernel/viengoos/projects.mdwn @@ -8,58 +8,10 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!tag open_issue_viengoos]] - Some projects: -# Minor - -## New hash function - -The current hash function in libhurd-ihash results in a lot of -collisions when the hash table is 80% full. To overcome this, we keep -hash tables at most 30% full. This represents a fair amount of -overhead. Find a better algorithm. There can either be one that is -appropriate in the general case or one that works well in a relevant, -specific case, e.g., viengoos/object.c uses a hash to find the object -corresponding to a frame, which is keyed on its physical address. - -# Major - -## Address Space Management - -In Viengoos, a process's address space is managed entirely in user -space by the process itself. This creates two interesting problems: -dealing with circular dependencies resulting from having to manage the -address space data structures and accessing and manipulating the -address space data structures. - -First, managing the address space requires resources, which in turn -may require address space (e.g., data structures require memory which -require address space, etc.). We currently break this circular -dependency by trying to keep enough resources in reserve that -allocating resources for managing the address space never requires -more resources than are minimally in the reserve. The reserve is -currently chosen in an ad-hoc fashion. It would be nice to determine -it more systematically. Moreover, it would be nice to reduce the -cases in which a reserve is required. This may be possible by -restructuring some of the code. - -Second, the address space data structures are protected using a single -lock. This not only means that only a single thread can be updating -the address space at a time, but that if a thread faults and the -address space is locked, then the process dead locks! It should be -possible to at least walk the address space using lock-free -techniques. This requires updating the address space construction -code such that all addresses remain valid during any given -manipulation. Second, to avoid the mentioned dead-lock problem, we -try to ensure that accessing the data structures will never result in -a fault. This means protecting the stack. An alternative approach is -to use undo buffers. - -# Thesis - -## Capability aware compiler - -Modify, e.g., gcc to understand capability semantics and teach gcc how -to optimize it, e.g., how to batch and combine calls. +[[!inline +pages="microkernel/viengoos/projects/* and !microkernel/viengoos/projects/*/*" +show=0 +feeds=no +actions=yes]] diff --git a/microkernel/viengoos/projects/address_space_management.mdwn b/microkernel/viengoos/projects/address_space_management.mdwn new file mode 100644 index 00000000..2d00e4f4 --- /dev/null +++ b/microkernel/viengoos/projects/address_space_management.mdwn @@ -0,0 +1,40 @@ +[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_viengoos]] + +In Viengoos, a process's address space is managed entirely in user +space by the process itself. This creates two interesting problems: +dealing with circular dependencies resulting from having to manage the +address space data structures and accessing and manipulating the +address space data structures. + +First, managing the address space requires resources, which in turn +may require address space (e.g., data structures require memory which +require address space, etc.). We currently break this circular +dependency by trying to keep enough resources in reserve that +allocating resources for managing the address space never requires +more resources than are minimally in the reserve. The reserve is +currently chosen in an ad-hoc fashion. It would be nice to determine +it more systematically. Moreover, it would be nice to reduce the +cases in which a reserve is required. This may be possible by +restructuring some of the code. + +Second, the address space data structures are protected using a single +lock. This not only means that only a single thread can be updating +the address space at a time, but that if a thread faults and the +address space is locked, then the process dead locks! It should be +possible to at least walk the address space using lock-free +techniques. This requires updating the address space construction +code such that all addresses remain valid during any given +manipulation. Second, to avoid the mentioned dead-lock problem, we +try to ensure that accessing the data structures will never result in +a fault. This means protecting the stack. An alternative approach is +to use undo buffers. diff --git a/microkernel/viengoos/projects/capability-aware_compiler.mdwn b/microkernel/viengoos/projects/capability-aware_compiler.mdwn new file mode 100644 index 00000000..b4e465d9 --- /dev/null +++ b/microkernel/viengoos/projects/capability-aware_compiler.mdwn @@ -0,0 +1,16 @@ +[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled +[[GNU Free Documentation License|/fdl]]."]]"""]] + +[[!tag open_issue_viengoos]] + +Modify, e.g., gcc to understand capability semantics and teach gcc how +to optimize it, e.g., how to batch and combine calls. + +This project is deemed suitable for a thesis. diff --git a/microkernel/viengoos/projects/new_hash_function.mdwn b/microkernel/viengoos/projects/new_hash_function.mdwn new file mode 100644 index 00000000..d0374720 --- /dev/null +++ b/microkernel/viengoos/projects/new_hash_function.mdwn @@ -0,0 +1,22 @@ +[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_viengoos]] + +The current hash function in libhurd-ihash results in a lot of +collisions when the hash table is 80% full. To overcome this, we keep +hash tables at most 30% full. This represents a fair amount of +overhead. Find a better algorithm. There can either be one that is +appropriate in the general case or one that works well in a relevant, +specific case, e.g., viengoos/object.c uses a hash to find the object +corresponding to a frame, which is keyed on its physical address. + +Note that this applies to the Hurd's [[hurd/libihash]], too. |