diff options
Diffstat (limited to 'microkernel/mach')
38 files changed, 2079 insertions, 383 deletions
diff --git a/microkernel/mach/concepts.mdwn b/microkernel/mach/concepts.mdwn index 04dbb1c6..0f7cbf00 100644 --- a/microkernel/mach/concepts.mdwn +++ b/microkernel/mach/concepts.mdwn @@ -1,6 +1,33 @@ -[[Mach]] is a first-generation [[microkernel]]. Mach's basic abstractions -include [[address_space]]s in the form of [[task]]s, execution contexts in the -form of [[thread]]s, [[IPC]], [[capabilities|capability]] in the form of [[port]]s, and -[[memory_object]]s, which enable Mach's [[external_pager_mechanism]]. +[[!meta copyright="Copyright © 2002, 2003, 2007, 2010 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[Mach]] is a first-generation [[microkernel]]. + +Mach's basic abstractions include [[virtual_address_space]]s in the form of +[[task]]s, execution contexts in the form of [[thread]]s, [[IPC]], +[[capabilities|capability]] in the form of [[port]]s, and [[memory_object]]s, +which enable Mach's [[external_pager_mechanism]]. + +Controlling [[task]]s, their [[virtual_address_space]], [[thread]]s, and other +system objects in Mach is implemented by using [[port]]s, as opposed to other +[[kernel]]s' [[system_call]] interface: almost all of the Mach API is +implemented by sending [[message]]s to [[port]]s. Device drivers that reside +in kernel space are controlled by ports, too. Mach's [[API]] is well-[[documented|documentation]]. + +[[!toggleable id=mach_kernel_principles text="""[[!template id=note +text="*[[mach\_kernel\_principles|documentation]]*: +{{$mach#kernel_principles}}"]]"""]] + +In particular the [[!toggle id=mach_kernel_principles +text="[mach\_kernel\_principles]"]] book further elaborates on Mach's concepts +and principles. diff --git a/microkernel/mach/continuation.mdwn b/microkernel/mach/continuation.mdwn new file mode 100644 index 00000000..7a3267f3 --- /dev/null +++ b/microkernel/mach/continuation.mdwn @@ -0,0 +1,24 @@ +[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[Mach]] internally uses *continuation*s for kernel [[thread]] management. + +The advantage is that not a full kernel thread stack has to be preserved in +case that a thread is about to enter a blocking state. This saves space. It +is not clear this is still worthwhile given today's RAM offerings. (How many +kernel threads are there, typically?) + +And, this would no longer be possible in case Mach were be made a +[[preemptive|preemtion]] kernel. In the latter case, the kernel itself, that +is, kernel threads can be preempted, and then their full state needs to be +preserved. + +[[!tag open_issue_documentation]] <!-- Not linked to from any Mach page. Move +to GNU Mach pages, as this is only an implementation detail? --> diff --git a/microkernel/mach/deficiencies.mdwn b/microkernel/mach/deficiencies.mdwn new file mode 100644 index 00000000..e1f6debc --- /dev/null +++ b/microkernel/mach/deficiencies.mdwn @@ -0,0 +1,522 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation open_issue_gnumach]] + + +# IRC, freenode, #hurd, 2012-06-29 + + <henrikcozza> I do not understand what are the deficiencies of Mach, the + content I find on this is vague... + <antrik> the major problems are that the IPC architecture offers poor + performance; and that resource usage can not be properly accounted to the + right parties + <braunr> antrik: the more i study it, the more i think ipc isn't the + problem when it comes to performance, not directly + <braunr> i mean, the implementation is a bit heavy, yes, but it's fine + <braunr> the problems are resource accounting/scheduling and still too much + stuff inside kernel space + <braunr> and with a very good implementation, the performance problem would + come from crossing address spaces + <braunr> (and even more on SMP, i've been thinking about it lately, since + it would require syncing mmu state on each processor currently using an + address space being modified) + <antrik> braunr: the problem with Mach IPC is that it requires too many + indirections to ever be performant AIUI + <braunr> antrik: can you mention them ? + <antrik> the semantics are generally quite complex, compared to Coyotos for + example, or even Viengoos + <braunr> antrik: the semantics are related to the message format, which can + be simplified + <braunr> i think everybody agrees on that + <braunr> i'm more interested in the indirections + <antrik> but then it's not Mach IPC anymore :-) + <braunr> right + <braunr> 22:03 < braunr> i mean, the implementation is a bit heavy, yes, + but it's fine + <antrik> that's not an implementation issue + <braunr> that's what i meant by heavy :) + <braunr> well, yes and no + <braunr> Mach IPC have changed over time + <braunr> it would be newer Mach IPC ... :) + <antrik> the fact that data types are (supposed to be) transparent to the + kernel is a major part of the concept, not just an implementation detail + <antrik> but it's not just the message format + <braunr> transparent ? + <braunr> but they're not :/ + <antrik> the option to buffer in the kernel also adds a lot of complexity + <braunr> buffer in the kernel ? + <braunr> ah you mean message queues + <braunr> yes + <antrik> braunr: eh? the kernel parses all the type headers during transfer + <braunr> yes, so it's not transparent at all + <antrik> maybe you have a different understanding of "transparent" ;-) + <braunr> i guess + <antrik> I think most of the other complex semantics are kinda related to + the in-kernel buffering... + <braunr> i fail to see why :/ + <antrik> well, it allows ports rights to be destroyed while a message is in + transfer. a lot of semantics revolve around what happens in that case + <braunr> yes but it doesn't affect performance a lot + <antrik> sure it does. it requires a lot of extra code and indirections + <braunr> not a lot of it + <antrik> "a lot" is quite a relative term :-) + <antrik> compared to L4 for example, it *is* a lot + <braunr> and those indirections (i think you refer to more branching here) + are taken only when appropriate, and can be isolated, improved through + locality, etc.. + <braunr> the features they add are also huge + <braunr> L4 is clearly insufficient + <braunr> all current L4 forks have added capabilities .. + <braunr> (that, with the formal verification, make se4L one of the + "hottest" recent system projects) + <braunr> seL4* + <antrik> yes, but with very few extra indirection I think... similar to + EROS (which claims to have IPC almost as efficient as the original L4) + <braunr> possibly + <antrik> I still fail to see much real benefit in formal verification :-) + <braunr> but compared to other problems, this added code is negligible + <braunr> antrik: for a microkernel, me too :/ + <braunr> the kernel is already so small you can simply audit it :) + <antrik> no, it's not neglible, if you go from say two cache lines touched + per IPC (original L4) to dozens (Mach) + <antrik> every additional variable that needs to be touched to resolve some + indirection, check some condition adds significant overhead + <braunr> if you compare the dozens to the huge amount of inter processor + interrupt you get each time you change the kernel map, it's next to + nothing .. + <antrik> change the kernel map? not sure what you mean + <braunr> syncing address spaces on hundreds of processors each time you + send a message is a real scalability issue here (as an example), where + Mach to L4 IPC seem like microoptimization + <youpi> braunr: modify, you mean? + <braunr> yes + <youpi> (not switchp + <youpi> ) + <braunr> but that's only one example + <braunr> yes, modify, not switch + <braunr> also, we could easily get rid of the ihash library + <braunr> making the message provide the address of the object associated to + a receive right + <braunr> so the only real indirection is the capability, like in other + systems, and yes, buffering adds a bit of complexity + <braunr> there are other optimizations that could be made in mach, like + merging structures to improve locality + <pinotree> "locality"? + <braunr> having rights close to their target port when there are only a few + <braunr> pinotree: locality of reference + <youpi> for cache efficiency + <antrik> hundreds of processors? let's stay realistic here :-) + <braunr> i am .. + <braunr> a microkernel based system is also a very good environment for RCU + <braunr> (i yet have to understand how liburcu actually works on linux) + <antrik> I'm not interested in systems for supercomputers. and I doubt + desktop machines will get that many independant cores any time soon. we + still lack software that could even romotely exploit that + <braunr> hum, the glibc build system ? :> + <braunr> lol + <youpi> we have done a survey over the nix linux distribution + <youpi> quite few packages actually benefit from a lot of cores + <youpi> and we already know them :) + <braunr> what i'm trying to say is that, whenever i think or even measure + system performance, both of the hurd and others, i never actually see the + IPC as being the real performance problem + <braunr> there are many other sources of overhead to overcome before + getting to IPC + <youpi> I completely agree + <braunr> and with the advent of SMP, it's even more important to focus on + contention + <antrik> (also, 8 cores aren't exactly a lot...) + <youpi> antrik: s/8/7/ , or even 6 ;) + <antrik> braunr: it depends a lot on the use case. most of the problems we + see in the Hurd are probably not directly related to IPC performance; but + I pretty sure some are + <antrik> (such as X being hardly usable with UNIX domain sockets) + <braunr> antrik: these have more to do with the way mach blocks than IPC + itself + <braunr> similar to the ext2 "sleep storm" + <antrik> a lot of overhead comes from managing ports (for for example), + which also mostly comes down to IPC performance + <braunr> antrik: yes, that's the main indirection + <braunr> antrik: but you need such management, and the related semantics in + the kernel interface + <braunr> (although i wonder if those should be moved away from the message + passing call) + <antrik> you mean a different interface for kernel calls than for IPC to + other processes? that would break transparency in a major way. not sure + we really want that... + <braunr> antrik: no + <braunr> antrik: i mean calls specific to right management + <antrik> admittedly, transparency for port management is only useful in + special cases such as rpctrace, and that probably could be served better + with dedicated debugging interfaces... + <braunr> antrik: i.e. not passing rights inside messages + <antrik> passing rights inside messages is quite essential for a capability + system. the problem with Mach IPC in regard to that is that the message + format allows way more flexibility than necessary in that regard... + <braunr> antrik: right + <braunr> antrik: i don't understand why passing rights inside messages is + important though + <braunr> antrik: essential even + <youpi> braunr: I guess he means you need at least one way to pass rights + <antrik> braunr: well, for one, you need to pass a reply port with each RPC + request... + <braunr> youpi: well, as he put, the message passing call is overpowered, + and this leads to many branches in the code + <braunr> antrik: the reply port is obvious, and can be optimized + <braunr> antrik: but the case i worry about is passing references to + objects between tasks + <braunr> antrik: rights and identities with the auth server for example + <braunr> antrik: well ok forget it, i just recall how it actually works :) + <braunr> antrik: don't forget we lack thread migration + <braunr> antrik: you may not think it's important, but to me, it's a major + improvement for RPC performance + <antrik> braunr: how can seL4 be the most interesting microkernel + then?... ;-) + <braunr> antrik: hm i don't know the details, but if it lacks thread + migration, something is wrong :p + <braunr> antrik: they should work on viengoos :) + <antrik> (BTW, AIUI thread migration is quite related to passive objects -- + something Hurd folks never dared seriously consider...) + <braunr> i still don't know what passive objects are, or i have forgotten + it :/ + <antrik> no own control threads + <braunr> hm, i'm still missing something + <braunr> what do you refer to by control thread ? + <braunr> with* + <antrik> i.e. no main loop etc.; only activated by incoming calls + <braunr> ok + <braunr> well, if i'm right, thomas bushnel himself wrote (recently) that + the ext2 "sleep" performance issue was expected to be solved with thread + migration + <braunr> so i guess they definitely considered having it + <antrik> braunr: don't know what the "sleep peformance issue" is... + <braunr> http://lists.gnu.org/archive/html/bug-hurd/2011-12/msg00032.html + <braunr> antrik: also, the last message in the thread, + http://lists.gnu.org/archive/html/bug-hurd/2011-12/msg00050.html + <braunr> antrik: do you consider having a reply port being an avoidable + overhead ? + <antrik> braunr: not sure. I don't remember hearing of any capability + system doing this kind of optimisation though; so I guess there are + reasons for that... + <braunr> antrik: yes me too, even more since neal talked about it on + viengoos + <antrik> I wonder whether thread management is also such a large overhead + with fully sync IPC, on L4 or EROS for example... + <braunr> antrik: it's still a very handy optimization for thread scheduling + <braunr> antrik: it makes solving priority inversions a lot easier + <antrik> actually, is thread scheduling a problem at all with a thread + activation approach like in Viengoos? + <braunr> antrik: thread activation is part of thread migration + <braunr> antrik: actually, i'd say they both refer to the same thing + <antrik> err... scheduler activation was the term I wanted to use + <braunr> same + <braunr> well + <braunr> scheduler activation is too vague to assert that + <braunr> antrik: do you refer to scheduler activations as described in + http://en.wikipedia.org/wiki/Scheduler_activations ? + <antrik> my understanding was that Viengoos still has traditional threads; + they just can get scheduled directly on incoming IPC + <antrik> braunr: that Wikipedia article is strange. it seems to use + "scheduler activations" as a synonym for N:M multithreading, which is not + at all how I understood it + <youpi> antrik: I used to try to keep a look at those pages, to fix such + wrong things, but left it + <braunr> antrik: that's why i ask + <antrik> IIRC Viengoos has a thread associated with each receive + buffer. after copying the message, the kernel would activate the + processes activation handler, which in turn could decide to directly + schedule the thead associated with the buffer + <antrik> or something along these lines + <braunr> antrik: that's similar to mach handoff + <youpi> antrik: generally enough, all the thread-related pages on wikipedia + are quite bogus + <antrik> nah, handoff just schedules the process; which is not useful, if + the right thread isn't activated in turn... + <braunr> antrik: but i think it's more than that, even in viengoos + <youpi> for instance, the french "thread" page was basically saying that + they were invented for GUIs to overlap computation with user interaction + <braunr> .. :) + <antrik> youpi: good to know... + <braunr> antrik: the "misunderstanding" comes from the fact that scheduler + activations is the way N:M threading was implemented on netbsd + <antrik> youpi: that's a refreshing take on the matter... ;-) + <braunr> antrik: i'll read the critique and viengoos doc/source again to be + sure about what we're talking :) + <braunr> antrik: as threading is a major issue in mach, and one of the + things i completely changed (and intend to change) in x15, whenever i get + to work on that again ..... :) + <braunr> antrik: interestingly, the paper about scheduler activations was + written (among others) by brian bershad, in 92, when he was actively + working on research around mach + <antrik> braunr: BTW, I have little doubt that making RPC first-class would + solve a number of problems... I just wonder how many others it would open + + +# IRC, freenode, #hurd, 2012-09-04 + +X15 + + <braunr> it was intended as a mach clone, but now that i have better + knowledge of both mach and the hurd, i don't want to retain mach + compatibility + <braunr> and unlike viengoos, it's not really experimental + <braunr> it's focused on memory and cpu scalability, and performance, with + techniques likes thread migration and rcu + <braunr> the design i have in mind is closer to what exists today, with + strong emphasis on scalability and performance, that's all + <braunr> and the reason the hurd can't be modified first is that my design + relies on some important design changes + <braunr> so there is a strong dependency on these mechanisms that requires + the kernel to exists first + + +## IRC, freenode, #hurd, 2012-09-06 + +In context of [[open_issues/multithreading]] and later [[open_issues/select]]. + + <gnu_srs> And you will address the design flaws or implementation faults + with x15? + <braunr> no + <braunr> i'll address the implementation details :p + <braunr> and some design issues like cpu and memory resource accounting + <braunr> but i won't implement generic resource containers + <braunr> assuming it's completed, my work should provide a hurd system on + par with modern monolithic systems + <braunr> (less performant of course, but performant, scalable, and with + about the same kinds of problems) + <braunr> for example, thread migration should be mandatory + <braunr> which would make client calls behave exactly like a userspace task + asking a service from the kernel + <braunr> you have to realize that, on a monolithic kernel, applications are + clients, and the kernel is a server + <braunr> and when performing a system call, the calling thread actually + services itself by running kernel code + <braunr> which is exactly what thread migration is for a multiserver system + <braunr> thread migration also implies sync IPC + <braunr> and sync IPC is inherently more performant because it only + requires one copy, no in kernel buffering + <braunr> sync ipc also avoids message floods, since client threads must run + server code + <gnu_srs> and this is not achievable with evolved gnumach and/or hurd? + <braunr> well that's not entirely true, because there is still a form of + async ipc, but it's a lot less likely + <braunr> it probably is + <braunr> but there are so many things to change i prefer starting from + scratch + <braunr> scalability itself probably requires a revamp of the hurd core + libraries + <braunr> and these libraries are like more than half of the hurd code + <braunr> mach ipc and vm are also very complicated + <braunr> it's better to get something new and simpler from the start + <gnu_srs> a major task nevertheless:-D + <braunr> at least with the vm, netbsd showed it's easier to achieve good + results from new code, as other mach vm based systems like freebsd + struggled to get as good + <braunr> well yes + <braunr> but at least it's not experimental + <braunr> everything i want to implement already exists, and is tested on + production systems + <braunr> it's just time to assemble those ideas and components together + into something that works + <braunr> you could see it as a qnx-like system with thread migration, the + global architecture of the hurd, and some improvements from linux like + rcu :) + + +### IRC, freenode, #hurd, 2012-09-07 + + <antrik> braunr: thread migration is tested on production systems? + <antrik> BTW, I don't think that generally increasing the priority of + servers is a good idea + <antrik> in most cases, IPC should actually be sync. slpz looked at it at + some point, and concluded that the implementation actually has a + fast-path for that case. I wonder what happens to scheduling in this case + -- is the receiver sheduled immediately? if not, that's something to + fix... + <braunr> antrik: qnx does something very close to thread migration, yes + <braunr> antrik: i agree increasing the priority isn't a good thing, but + it's the best of the quick and dirty ways to reduce message floods + <braunr> the problem isn't sync ipc in mach + <braunr> the problem is the notifications (in our cases the dead name + notifications) that are by nature async + <braunr> and a malicious program could send whatever it wants at the + fastest rate it can + <antrik> braunr: malicious programs can do any number of DOS attacks on the + Hurd; I don't see how increasing priority of system servers is relevant + in that context + <antrik> (BTW, I don't think dead name notifications are async by + nature... just like for most other IPC, the *usual* case is that a server + thread is actively waiting for the message when it's generated) + <braunr> antrik: it's async with respect to the client + <braunr> antrik: and malicious programs shouldn't be able to do that kind + of dos + <braunr> but this won't be fixed any time soon + <braunr> on the other hand, a higher priority helps servers not create too + many threads because of notifications, and that's a good thing + <braunr> gnu_srs: the "fix" for this will be to rewrite select so that it's + synchronous btw + <braunr> replacing dead name notifications with something like cancelling a + previously installed select request + <antrik> no idea what "async with respect to the client" means + <braunr> it means the client doesn't wait for anything + <antrik> what is the client? what scenario are you talking about? how does + it affect scheduling? + <braunr> for notifications, it's usually the kernel + <braunr> it doesn't directly affect scheduling + <braunr> it affects the amount of messages a hurd server has to take care + of + <braunr> and the more messages, the more threads + <braunr> i'm talking about event loops + <braunr> and non blocking (or very short) selects + <antrik> the amount of messages is always the same. the question is whether + they can be handled before more come in. which would be the case if be + default the receiver gets scheduled as soon as a message is sent... + <braunr> no + <braunr> scheduling handoff doesn't imply the thread will be ready to + service the next message by the time a client sends a new one + <braunr> the rate at which a message queue gets filled has nothing to do + with scheduling handoff + <antrik> I very much doubt rates come into play at all + <braunr> well they do + <antrik> in my understanding the problem is that a lot of messages are sent + before the receive ever has a chance to handle them. so no matter how + fast the receiver is, it looses + <braunr> a lot of non blocking selects means a lot of reply ports + destroyed, a lot of dead name notifications, and what i call message + floods at server side + <braunr> no + <braunr> it used to work fine with cthreads + <braunr> it doesn't any more with pthreads because pthreads are slightly + slower + <antrik> if the receiver gets a chance to do some work each time a message + arrives, in most cases it would be free to service the next request with + the same thread + <braunr> no, because that thread won't have finished soon enough + <antrik> no, it *never* worked fine. it might have been slighly less + terrible. + <braunr> ok it didn't work fine, it worked ok + <braunr> it's entirely a matter of rate here + <braunr> and that's the big problem, because it shouldn't + <antrik> I'm pretty sure the thread would finish before the time slice ends + in almost all cases + <braunr> no + <braunr> too much contention + <braunr> and in addition locking a contended spin lock depresses priority + <braunr> so servers really waste a lot of time because of that + <antrik> I doubt contention would be a problem if the server gets a chance + to handle each request before 100 others come in + <braunr> i don't see how this is related + <braunr> handling a request doesn't mean entirely processing it + <braunr> there is *no* relation between handoff and the rate of incoming + message rate + <braunr> unless you assume threads can always complete their task in some + fixed and low duration + <antrik> sure there is. we are talking about a single-processor system + here. + <braunr> which is definitely not the case + <braunr> i don't see what it changes + <antrik> I'm pretty sure notifications can generally be handled in a very + short time + <braunr> if the server thread is scheduled as soon as it gets a message, it + can also get preempted by the kernel before replying + <braunr> no, notifications can actually be very long + <braunr> hurd_thread_cancel calls condition_broadcast + <braunr> so if there are a lot of threads on that .. + <braunr> (this is one of the optimizations i have in mind for pthreads, + since it's possible to precisely select the target thread with a doubly + linked list) + <braunr> but even if that's the case, there is no guarantee + <braunr> you can't assume it will be "quick enough" + <antrik> there is no guarantee. but I'm pretty sure it will be "quick + enough" in the vast majority of cases. which is all it needs. + <braunr> ok + <braunr> that's also the idea behind raising server priorities + <antrik> braunr: so you are saying the storms are all caused by select(), + and once this is fixed, the problem should be mostly gone and the + workaround not necessary anymore? + <braunr> yes + <antrik> let's hope you are right :-) + <braunr> :) + <antrik> (I still think though that making hand-off scheduling default is + the right thing to do, and would improve performance in general...) + <braunr> sure + <braunr> well + <braunr> no it's just a hack ;p + <braunr> but it's a right one + <braunr> the right thing to do is a lot more complicated + <braunr> as roland wrote a long time ago, the hurd doesn't need dead-name + notifications, or any notification other than the no-sender (which can be + replaced by a synchronous close on fd like operation) + <antrik> well, yes... I still think the viengoos approach is promising. I + meant the right thing to do in the existing context ;-) + <braunr> better than this priority hack + <antrik> oh? you happen to have a link? never heard of that... + <braunr> i didn't want to do it initially, even resorting to priority + depression on trhead creation to work around the problem + <braunr> hm maybe it wasn't him, i can't manage to find it + <braunr> antrik: + http://lists.gnu.org/archive/html/l4-hurd/2003-09/msg00009.html + <braunr> "Long ago, in specifying the constraints of + <braunr> what the Hurd needs from an underlying IPC system/object model we + made it + <braunr> very clear that we only need no-senders notifications for object + <braunr> implementors (servers)" + <braunr> "We don't in general make use of dead-name notifications, + <braunr> which are the general kind of object death notification Mach + provides and + <braunr> what serves as task death notification." + <braunr> "In the places we do, it's to serve + <braunr> some particular quirky need (and mostly those are side effects of + Mach's + <braunr> decouplable RPCs) and not a semantic model we insist on having." + + +### IRC, freenode, #hurd, 2012-09-08 + + <antrik> The notion that seemed appropriate when we thought about these + issues for + <antrik> Fluke was that the "alert" facility be a feature of the IPC system + itself + <antrik> rather than another layer like the Hurd's io_interrupt protocol. + <antrik> braunr: funny, that's *exactly* what I was thinking when looking + at the io_interrupt mess :-) + <antrik> (and what ultimately convinced me that the Hurd could be much more + elegant with a custom-tailored kernel rather than building around Mach) + + +## IRC, freenode, #hurd, 2012-09-24 + + <braunr> my initial attempt was a mach clone + <braunr> but now i want a mach-like kernel, without compability + <lisporu> which new licence ? + <braunr> and some very important changes like sync ipc + <braunr> gplv3 + <braunr> (or later) + <lisporu> cool 8) + <braunr> yes it is gplv2+ since i didn't take the time to read gplv3, but + now that i have, i can't use anything else for such a project: ) + <lisporu> what is mach-like ? (how it is different from Pistachio like ?) + <braunr> l4 doesn't provide capabilities + <lisporu> hmmm.. + <braunr> you need a userspace for that + <braunr> +server + <braunr> and it relies on complete external memory management + <lisporu> how much work is done ? + <braunr> my kernel will provide capabilities, similar to mach ports, but + simpler (less overhead) + <braunr> i want the primitives right + <braunr> like multiprocessor, synchronization, virtual memory, etc.. + + +### IRC, freenode, #hurd, 2012-09-30 + + <braunr> for those interested, x15 is now a project of its own, with no + gnumach compability goal, and covered by gplv3+ diff --git a/microkernel/mach/discussion.mdwn b/microkernel/mach/discussion.mdwn deleted file mode 100644 index 589e302d..00000000 --- a/microkernel/mach/discussion.mdwn +++ /dev/null @@ -1,23 +0,0 @@ -## <a name="Maintenance_of_the_Mach_web"> Maintenance of the Mach web </a> - -**_Old discussions:_** [[WIKIHOMEURLMachTOPICrev13]] - -Interesting, for consistency sake I'll think about making your changes you made on the right hand side to the other web WebHome pages. I guess it's not critical that they are identical, but I was trying to keep them identical if possible. I also wanted it to be "light" enough feature wise that it doesn't overpower the page. You've added back a few of the features, so we obviously differ in how important you and I think these features are. That's OK, I'll think about it some more and we'll see what happens. - -Oh, I see you added back [[WebTopicList]] and [[WebPreferences]]. I purposely removed [[WebPreferences]] from the lists on the right because it has nothing to do with navigation. I also didn't think that people actually use topic names to navigate. If they do they could search for them. Keeping the number to four items instead of six and keeping the descriptions concise makes a big difference when I view the page. - -(goes off to think more...) - -and eat... ;-) - --- [[Main/GrantBow]] - 29 Dec 2002 - -**_Reasons for my change:_** - -1. [[WebTopicList]] is a lot quicker than the [[WebIndex]] - brings down the load times and the load of the server -2. [[WebPreferences]] - users might be curious to see what can be modified. Changes should of course only be made in their home topics, like in %WIKIUSERNAME%. However, the [[WebPreferences]] can serve as an inspiration. Therefore we should perhaps make sure only the [[Main/TWikiAdminGroup]] members can alter the \*Preferences topics. -3. If you look closely I've also reordered the links. Shorter names first and long ones last, I tried to keep the descriptions brief and in proportional length as well. - -I don't know about you, but keeping the number of items to four rather than six doesn't really matter to me. The text is quite small and if it's the space we're after the [[WebStatistics]] does take up more than the navigation links. - --- [[Main/JoachimNilsson]] - 29 Dec 2002 diff --git a/microkernel/mach/documentation.mdwn b/microkernel/mach/documentation.mdwn index 3b12bfac..cc880ab6 100644 --- a/microkernel/mach/documentation.mdwn +++ b/microkernel/mach/documentation.mdwn @@ -1,24 +1,33 @@ -[[!meta copyright="Copyright © 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008 -Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, +2010 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] - - [Meet Mach](http://www.stepwise.com/Articles/Technical/MeetMach.html), a - summary of Mach's history and main concepts. + * Mach's [[concepts]]. + + * [*Meet Mach* by James + Scott](http://beefchunk.com/documentation/macosx-programming/Meet_Mach.pdf), + a summary of Mach's history and main concepts. * *[[The_GNU_Mach_Reference_Manual|gnumach/reference_manual]]*. - - OSF's [Kernel Interface (ps)](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/kernel_interface.ps) - [Kernel Interface (pdf)](http://shakthimaan.com/downloads/hurd/kernel_interface.pdf) + * {{$mach#kernel_foundation_unix}} + + * {{$mach#vm}} + + * {{$mach#kernel_principles}} + + * {{$mach#kernel_interface}} + + * {{$mach#server_writer}} - - OSF's [Kernel Principles (ps)](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/kernel_principles.ps) - [Kernel Principles (pdf)](http://shakthimaan.com/downloads/hurd/kernel_principles.pdf) + * {{$mach#server_interface}} * [*The Unofficial GNU Mach IPC beginner's guide*](http://hurdextras.nongnu.org/ipc_guide/), an easy introduction to diff --git a/microkernel/mach/external_pager_mechanism.mdwn b/microkernel/mach/external_pager_mechanism.mdwn index b175d1cc..05a6cc56 100644 --- a/microkernel/mach/external_pager_mechanism.mdwn +++ b/microkernel/mach/external_pager_mechanism.mdwn @@ -1,92 +1,93 @@ -[[!meta copyright="Copyright © 2002, 2007, 2008 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2002, 2007, 2008, 2010, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -Mach provides a so-called external pager [[mechanism]]. This +Mach provides a so-called *external pager [[mechanism]]*. This mechanism serves to separate *managing memory* from *managing -content*. Mach does the former while user space tasks do the +content*. Mach does the former while user-space processes do the latter. +[[!tag open_issue_documentation]] <!-- Should probably refer to {{$mach#vm}}. +--> + # Introduction -In Mach, a task's [[Mach/AddressSpace]] consists of references -to [[Mach/MemoryObjects]]. A memory object is designated using -a [[port]] (a port is just a [[capability]]) and -implemented by a normal process. +In Mach, a [[task]]'s [[virtual_address_space]] consists of references to +[[memory_object]]s. To associate a memory object with a portion of a task's -address space, vm\_map is invoked a capability designating +address space, `vm_map` is invoked on a capability designating the task and passing a reference to the memory object and the offset at which to install it. (The first time a task maps an object, Mach sends an initialization message to the server including a control capability, which it uses to supply pages to the kernel.) This is essentially -the same as mapping a file into an address space on [[Unix]] -using mmap. +the same as mapping a file into an address space on [[UNIX]] +using `mmap`. -When a task faults, Mach checks to see if there is a memory +When a task [[faults|page_fault]], Mach checks to see if there is a memory object associated with the fault address. If not, the task -is sent an exception, which is normally further propagated +is sent an [[exception]], which is normally further propagated as a segmentation fault. If there is an associated memory -object, Mach checks whether the corresponding page is in core. -If it is, it installs the page and resumes the task. Mach -then invokes the memory object with the memory\_object\_request +object, Mach checks whether the corresponding [[page]] is in core. +If it is, it installs the page and resumes the task. Mach +then invokes the memory object with the `memory_object_request` method and the page to read. The memory manager then fetches or creates the content as appropriate and supplies it to -Mach using the memory\_object\_supply method. +Mach using the `memory_object_supply` method. # Creating and Mapping a Memory Object The following illustrates the basic idea: -> ________ -> / \ -> | Mach | -> \________/ -> /| / |\ \ -> (C) vm_map / / m_o_ready (E)\ \ (D) memory_object_init -> / |/ (F) return \ \| -> ________ ________ -> / \ -----> / \ -> | Client | (A) open | Server | -> \________/ <----- \________/ -> (B) memory_object - -(A) The client sends an "open" rpc to the server. + ________ + / \ + | Mach | + \________/ + /| / |\ \ + (C) vm_map / / m_o_ready (E)\ \ (D) memory_object_init + / |/ (F) return \ \| + ________ ________ + / \ -----> / \ + | Client | (A) open | Server | + \________/ <----- \________/ + (B) memory_object + +(A) The client sends an `open` [[RPC]] to the server. (B) The server creates a memory object (i.e., a port receive right), adds it to the port set that it is listening on and returns a capability (a port send right) to the client. (C) The client attempts to map the object into its address space using -the vm\_map rpc. It passes a reference to the port that the server gave +the `vm_map` RPC. It passes a reference to the port that the server gave it to the vm server (typically Mach). (D) Since Mach has never seen the object before, it queues a -memory\_object\_init on the given port along with a send right (the +`memory_object_init` on the given port along with a send right (the memory control port) for the manager to use to send messages to the kernel and also as an authentication mechanism for future interactions: the port is supplied so that the manager will be able to -identify from which kernel a given memory\_object\_* IPC is from. +identify from which kernel a given `memory_object_*` IPC is from. (E) The server dequeues the message, initializes internal data structures to manage the mapping and then invokes the -memory\_object\_ready method on the control object. +`memory_object_ready` method on the control object. (F) The kernel sees that the manager is ready, sets up the appropriate -mappings in the client and then replies to the vm\_map rpc indicating +mappings in the client's address space and then replies to the `vm_map` RPC indicating success. -There is nothing stopping others from playing "the kernel." This is +There is nothing stopping others from playing *the kernel*. This is not a security problem: clients must [[trust]] the server from whom they obtain memory objects and also the servers with whom they share the object. Multiple memory managers are a reality that should be @@ -96,37 +97,37 @@ mappings etc. # Resolving Page Faults -> (G) Client ________ -> resumed / \ -> | Mach | -> (A) Fault +----|------+ | \ (B) m_o_request (C) store_read -> ____|___ \_____|__/ |\ \| ________ _________ -> / +---\-------+ \ / \ / \ -> | Client | (F) | Server |<===>| storeio | -> \________/ m_o_supply \________/ \_________/ -> (E) return data | ^ -> | | (D) device_read -> v | -> ________ -> / Device \ -> | Driver | -> \________/ -> | ^ -> | | -> v -> ____________ -> / Hardware \ - -(A) The client does a memory access and faults. The kernel catches + (G) Client ________ + resumed / \ + | Mach | + (A) Fault +----|------+ | \ (B) m_o_request (C) store_read + ____|___ \_____|__/ |\ \| ________ _________ + / +---\-------+ \ / \ / \ + | Client | (F) | Server |<===>| storeio | + \________/ m_o_supply \________/ \_________/ + (E) return data | ^ + | | (D) device_read + v | + ________ + / Device \ + | Driver | + \________/ + | ^ + | | + v + ____________ + / Hardware \ + +(A) The client does a memory access and [[faults|page_fault]]. The kernel catches the fault and maps the address to the appropriate memory object. It -then invokes the memory\_object\_request method on the associated +then invokes the `memory_object_request` method on the associated capability. (In addition to the page to supply, it also supplies the control port so that the server can determine which kernel sent the message.) -(B) The manager dequeues the message. On the Hurd, this is translated -into a store\_read: a function in the libstore library which is used to -transparently manage block devices. The storeio server starts off as +(B) The manager dequeues the message. On the [[Hurd]], this is translated +into a `store_read`: a function in the [[hurd/libstore]] library which is used to +transparently manage block devices. The [[hurd/storeio]] server starts off as a separate process, however, if the server has the appropriate permission, the backing object can be contacted directly by the server. This layer of indirection is desirable when, for instance, a @@ -134,37 +135,37 @@ storeio running as root may want to only permit read only access to a resource, yet it cannot safely transfer its handle to the client. In this case, it would proxy the requests. -(C) The storeio server contacts, for instance, a device driver to do +(C) The storeio server contacts, for instance, a [[device_driver]] to do the read. This could also be a network block device (the NBD server in GNU/Linux), a file, a memory object, etc. -(D) The device driver allocates an anonymous page from the default -pager and reads the data into it. Once all of the operations are +(D) The device driver allocates an [[anonymous_page]] from the +[[default_pager]] and reads the data into it. Once all of the operations are complete, the device returns the data to the client unmapping it from its own address space at the same time. -(E) The storeio transfers the page to the server. The page is still +(E) The storeio server transfers the page to the server. The page is still anonymous. -(F) The manager does a memory\_object\_supply transferring the page to +(F) The manager does a `memory_object_supply` transferring the page to the kernel. Only now is the page not considered to be anonymous but managed. (G) The kernel caches the page, installs it in the client's virtual -address space and finally, resumes the client. +[[address_space]] and finally, resumes the client. # Paging Data Out -> Change manager Pager m_o_return store_write -> \ _________ (B) __(A)__ (C) ________ (D) _______ -> S | / Default \ / \ / \ / \ -> W |<=>| Pager |<=>| Mach |==>| server |<=>| storeio |<=> -> A | \_________/ \________/ \________/ \_______/ -> P | -> / + Change manager Pager m_o_return store_write + \ _________ (B) __(A)__ (C) ________ (D) _______ + S | / Default \ / \ / \ / \ + W |<=>| Pager |<=>| Mach |==>| server |<=>| storeio |<=> + A | \_________/ \________/ \________/ \_______/ + P | + / -(A) The paging [[policy]] is implemented by Mach: servers just implement +(A) The [[paging]] [[policy]] is implemented by Mach: servers just implement the [[mechanism]]. (B) Once the kernel has selected a page that it would like to evict, it @@ -173,10 +174,22 @@ if the server does not deallocate the page quickly enough, it cannot cause a denial of service: the kernel will just later double page it to swap (the default pager is part of the [[tcb]]). -(C) Mach then invokes memory\_object\_return method on the control -object. The server is expected to save the page free it in a timely +(C) Mach then invokes `memory_object_return` <!-- doesn't exist --> method on the control +object. The server is expected to save the page free <!-- ? --> it in a timely fashion. The server is not required to send a response to the kernel. -(D) The manager then transfers the data to the storeio which +(D) The manager then transfers the data to the storeio server which eventually sends it to disk. The device driver consumes the memory -doing the equivalent of a vm\_deallocate. +doing the equivalent of a `vm_deallocate`. + + +# Issues + + * [[open_issues/performance/io_system/read-ahead]] + + * [[open_issues/performance/io_system/clustered_page_faults]] + + +# GNU Hurd Usage + +Read about the [[Hurd's I/O path|hurd/io_path]]. diff --git a/microkernel/mach/gnumach.mdwn b/microkernel/mach/gnumach.mdwn index f3d6d5f9..edd0cfdb 100644 --- a/microkernel/mach/gnumach.mdwn +++ b/microkernel/mach/gnumach.mdwn @@ -1,15 +1,18 @@ -[[!meta copyright="Copyright © 2001, 2002, 2007, 2008 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2001, 2002, 2007, 2008, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -GNU Mach is the microkernel that the [[GNU_Hurd|hurd]] system is based on. +GNU Mach is the microkernel upon which a GNU Hurd system is based. It provides +an Inter Process Communication (IPC) mechanism that the Hurd uses to define +interfaces for implementing in a distributed multi-server fashion the services +a traditional operating system kernel provides. It is maintained by the Hurd developers for the GNU project and remains compatible with [[Mach]] 3.0. @@ -75,6 +78,7 @@ GNU/Hurd. * [[Building]] * [[Debugging]] * [[Boot_Trace]] + * [[Memory_Management]] * [[Projects]] * [[Rules]] * [[Open Issues|tag/open_issue_gnumach]] diff --git a/microkernel/mach/gnumach/boot_trace.mdwn b/microkernel/mach/gnumach/boot_trace.mdwn index d33ef25a..1badf712 100644 --- a/microkernel/mach/gnumach/boot_trace.mdwn +++ b/microkernel/mach/gnumach/boot_trace.mdwn @@ -1,12 +1,13 @@ -[[!meta copyright="Copyright © 2007, 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2007, 2008, 2011 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] `if NCPUS > 1` stuff is not being considered so far. @@ -215,6 +216,12 @@ is included in the section entitled >> kern/bootstrap.c: bootstrap\_create +>>> The [[grub/multiboot]] modules have been put somewhere into memory by +>>> [[GRUB]]. The boot scripts are parsed. The modules' ELF image's `PT_LOAD` +>>> sections are \`\`read'' (that is, `vm_allocate` and `copyout`) and turned +>>> into real [[task]]s. The multiboot modules' memory regions can be +>>> deallocated then. + >> [...] >> vm\_pageout diff --git a/microkernel/mach/gnumach/building.mdwn b/microkernel/mach/gnumach/building.mdwn index 9c075600..427fb083 100644 --- a/microkernel/mach/gnumach/building.mdwn +++ b/microkernel/mach/gnumach/building.mdwn @@ -1,5 +1,13 @@ -Additional to the following text, a further [[example]] has be posted. +[[!meta copyright="Copyright © 2006, 2007, 2008, 2011 Free Software Foundation, +Inc."]] +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] # Building [[GNU_Mach|gnumach]] from Source @@ -11,55 +19,32 @@ enabled) is around 50 MiB. ## Getting the Source Code -### Developers's RCS +You can either use the git repository (see <http://git.savannah.gnu.org/cgit/hurd/>), -See <http://savannah.gnu.org/cvs/?group=hurd>. + $ git clone http://git.savannah.gnu.org/cgit/hurd/gnumach.git/ - $ cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/sources/hurd co -r gnumach-1-branch gnumach - -(Most probably you want to get hold of the *GNU Mach 1 branch* and not the -trunk, which is also what we've done above.) - -You then have to create the automatically generatable files: - - $ ( cd gnumach && autoreconf --install ) - -### What Debian is currently using - -See [here](http://packages.debian.net/source/unstable/gnumach). +... or get the Debian sources, if you're using Debian. (See +[here](http://packages.debian.net/source/unstable/gnumach).) $ apt-get source gnumach Please see the Debian [[running/debian/FAQ]] before using `apt-get source`. -## Preparing for the Build +## On Debian Systems: -### ... on Debian systems +### Preparing for the Build -Building GNU Mach requires the *build-essential* and *fakeroot* packages, their -dependencies and additional packages that are specified by the source gnumach -package: +Building GNU Mach requires the *build-essential* and *fakeroot* packages, +and some additional dependencies specified by the gnumach source package: # apt-get install build-essential fakeroot # apt-get build-dep gnumach -### ... on non-Debian systems - -Apart from the case that you only want to install GNU Mach's header files (see -below), building GNU Mach requires you to have the Mach Interface Generator -installed. See [[building_MIG|mig/gnu_mig/building]] about how to do that, then come -back here. +### Building and Installing ... Debian `.deb` files -Additionally, building GNU Mach requires a C compiler, a standard C library and -your favourite flavor of awk (gawk) and GNU make. +Change into the directory with the downloaded / unpacked GNU Mach sources, -## Building and Installing - -### ... Debian `.deb` files - -Change into the directory with the downloaded / unpacked GNU Mach sources, e.g. - - $ cd gnumach-20050801 + $ cd gnumach-XXXXXXXX Start the build process with @@ -67,48 +52,68 @@ Start the build process with [[GNU_Mach|gnumach]] is now building. To use the new kernel, you must install the resulting `.deb` package which is located one directory above the build -directory and has a similar name as the build directory, e.g. +directory and has a similar name as the build directory: - # dpkg -i ../gnumach_20050801-4_hurd-i386.deb + # dpkg -i ../gnumach_XXXXXXXX-X_hurd-i386.deb You can now reboot your computer and enjoy the new kernel. -### [TODO] +## On non-Debian Systems: -GNU Mach should be built in a separate directory: +### Preparing for the Build - $ mkdir gnumach-build - $ cd gnumach-build +Building GNU Mach requires a C compiler, a _static_ 32 bit standard C library, +your favourite flavor of awk (gawk) and GNU make. -Find the path to your GNU Mach sources (`[...]/gnumach-1-branch`) and configure -it: +First, create the configuartion files: - $ [...]/gnumach-1-branch/configure [TODO] + $ cd gnumach + $ autoreconf --install -Build the kernel image: +GNU Mach (and the associated headers) need be built in a separate build directory: - $ make gnumach.gz + $ mkdir build + $ cd build -Optionally run the (tiny) test suite: +Run configure: - $ make check + $ ../configure --prefix= -You can then install and use `gnumach.gz`. +If building on a 64 bit host system, +you need a number of additional settings to force a 32 bit build: -[TODO.] + $ CPP='gcc -m32 -E -x c -undef -ansi' CC='gcc -m32' LD='ld -melf_i386' ../configure --prefix= --host=i686-unknown-linux-gnu -### Installing only the Header Files +### Installing the Header Files First -GNU Mach should be built in a separate directory: +In order to build GNU Mach, you will need a working MIG. +Building MIG in turn requires the GNU Mach header files to be already present. +So for bootstrapping MIG, you have to install the Mach headers first, +for example into `~/gnu/include/`: - $ mkdir gnumach-build - $ cd gnumach-build + $ make DESTDIR=~/gnu install-data -Find the path to your GNU Mach sources (`[...]/gnumach-1-branch`) and configure -it: +Now you can [[build_MIG|mig/gnu_mig/building]]. +Once you are done with that, come back here to finish the Mach build. - $ [...]/gnumach-1-branch/configure --prefix= +### Building and Installing -Install the header files into e.g. `~/gnu/include/`: +With MIG present, now build the kernel image: + + $ make gnumach.gz + +Optionally run the (tiny) test suite: + + $ make check + +It's a good idea to make a backup of the previously installed kernel, in case +you can't boot using the new one. That way, you can restore it after booting +from a rescue media (or mounting the disk image used by your vm). + + # cp /boot/gnumach.gz /boot/gnumach.gz.bak + +GNU Mach can now be moved into place, typically `/boot/gnumach.gz`, so that you +can boot your system with the new kernel. + + # cp gnumach.gz /boot - $ make DESTDIR=~/gnu install-data diff --git a/microkernel/mach/gnumach/building/example.mdwn b/microkernel/mach/gnumach/building/example.mdwn deleted file mode 100644 index 7db98547..00000000 --- a/microkernel/mach/gnumach/building/example.mdwn +++ /dev/null @@ -1,54 +0,0 @@ -[[!meta copyright="Copyright © 2007, 2008 Free Software Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] - -## Compiling GNU Mach microkernel - -Host development system is IBM T41 running Debian Sarge 3.1r0a GNU/Linux. - -* gcc version: 3.3.5 -* GNU sed version: 4.1.2 -* GNU make version: 3.8 -* mig version: 1.3-4 - -Obtained gnumach-1-branch sources from cvs: - - export CVS_RSH="ssh" - cvs -z3 -d:ext:anoncvs@ savannah.gnu.org:/cvsroot/hurd co -r gnumach-1-branch gnumach - -Obtained mig_1.3-4_i386.deb from -http://www.hadrons.org/~guillem/debian/pool/main/mig/. Installed it using dpkg: - - dpkg -i mig_1.3-4_i386.deb - -Entered into the gnumach sources and did the following for compilation: - - mkdir build - cd build - ../configure --host=i386-unknown-gnu0.2 --build=i586-pc-linux-gnu \ - --enable-kdb --enable-ide - make - -The kernel file is created in the build directory. Move it to /boot on the -testing x86 system Hurd partition. Rename it as gnumach1 and compress it: - - mv kernel gnumach1 - gzip gnumach1 - -Add a new entry on the testing machine /boot/grub/menu.lst to boot the new -kernel. - - title GNU Hurd K10 Compiled gnumach - kernel (hd0,3)/boot/gnumach1.gz root=device:hd2s4 -s - module (hd0,3)/hurd/ext2fs.static--multiboot-command-line=${kernel-command-line} \\ - --host-priv-port=${host-port} --device-master-port=${device-port} \\ - --exec-server-task=${exec-task} -T typed ${root} $(task-create)$(task-resume) - module (hd0,3)/lib/ld.so.1 /hurd/exec $(exec-task=task-create) - -Reboot into the new compiled mygnumach1.gz kernel! diff --git a/microkernel/mach/gnumach/debugging.mdwn b/microkernel/mach/gnumach/debugging.mdwn index 3a93c6ad..71e92459 100644 --- a/microkernel/mach/gnumach/debugging.mdwn +++ b/microkernel/mach/gnumach/debugging.mdwn @@ -1,23 +1,82 @@ -[[!meta copyright="Copyright © 2007, 2008, 2009 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2007, 2008, 2009, 2011, 2012 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +Here are some hints to debug with GNU Mach. + +[[!toc levels=2]] + + +# Kernel Debugger Mach has a built-in kernel debugger. [Manual](http://www.gnu.org/software/hurd/gnumach-doc/Kernel-Debugger.html). +First, make sure to enable it. Either by using a pre-packaged gnumach-image-something-dbg, or by passing --enable-kdb to the ./configure invocation. + +Then, reproduce the issue again. If something like a kernel trap happens, you will end up in the GNU Mach debugger. Otherwise, type control-alt-d to make Mach enter it by hand. + +If you are running in kvm or qemu, it is convenient to use the curses frontend to be able to copy/paste. + +To get the register values, type + + show registers + +To get a backtrace, type + + trace + +, which will print both function return addresses and function parameters, such as + + 0x107cf1(8088488,5e,40000008,2aa008,0) + 0x1071bc(0,0,0,0,0) + 0x106831(24fe00,2000,b,800,0) + +Run the addr2line tool on the return addresses: + + $ addr2line -i -f -e /boot/gnumach 0x107cf1 0x1071bc 0x106831 + +This will print the source code lines of the backtrace. + +To examine the backtrace of some given thread, use + + show all thread/u + +to get the whole listing of all tasks and threads. You can then use trace/t to trace a specific thread. + +Unfortunately, userland and kernelland use the same range of addresses, so one can not get userland traces easily. The Xen port uses different ranges, and in that case one can use trace/u to also get the userland trace. + +To examine a variable, use nm /boot/gnumach to get the address of the variable (e.g. 0x123400), and use + + x 0x123400 + +to read it. One can also write to it by using + + w 0x123400 + +Another interesting feature is watching a variable, by using + + watch 0x123400 + +and then type continue, to let Mach continue execution. The debugger will be entered again on any change in that variable. The watch is implemented in hardware, so it does not disturb or slow down execution at all. + + +# GDB in QEMU When you're [[running_a_system_in_QEMU|hurd/running/qemu]] you can directly [use GDB on the running kernel](http://www.nongnu.org/qemu/qemu-doc.html#SEC48). +# Code Inside the Kernel + Alternatively you can use an approach like this one: add the following code snippet to `device/ds_routines.c`'s `ds_device_open` function, right at the top of the function, and modify the code as needed. @@ -56,6 +115,8 @@ This is especially useful if you need to manually trigger some stuff inside the running kernel, as with the *D1* example. +## Writing to the Screen Buffer + If you're doing real low level debugging, you might want to put variations of the following snipped into the code, this code will write a `#` character at line `[LINE]`, column `[COLUMN]` on the screen: @@ -67,3 +128,22 @@ The call of `halt_cpu` will -- as the name suggests -- halt the system afterwards. This might be what you want or it might not, but it is needed at some place when running the kernel inside QEMU, as QEMU somehow decides not to update its display buffer anymore under certain conditions. + + +# Halting the CPU and Examining Registers + +IRC, freenode, #hurd, 2011-07-14: + + <braunr> one ugly trick i use when printf isn't available is to halt the + cpu + <braunr> then use info registers to know where the cpu is halted + <braunr> and you'll know if you reached that code or not + <braunr> (info registers is a qemu command) + + +# Serial Console + +IRC, freenode, #hurd, 2011-11-13: + + <youpi> use console=com0 + <youpi> to activate the console on the first serial port diff --git a/microkernel/mach/gnumach/hardware_compatibility_list.mdwn b/microkernel/mach/gnumach/hardware_compatibility_list.mdwn index 2152c079..874f5f07 100644 --- a/microkernel/mach/gnumach/hardware_compatibility_list.mdwn +++ b/microkernel/mach/gnumach/hardware_compatibility_list.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2009 Free Software Foundation, +[[!meta copyright="Copyright © 2007, 2008, 2009, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -6,8 +6,8 @@ id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] # CPU Architecture @@ -29,7 +29,7 @@ Read about further [[ports]]. # Memory -GNU Mach will use a maximum of 1 GiB of RAM. If your system has more, +GNU Mach will use a maximum of 1.7 GiB of RAM. If your system has more, the surplus will silently be ignored. (In past times, this would hinder GNU Mach from booting at all, but this has been fixed, so you no longer need to apply GRUB's `uppermem` directive.) @@ -68,10 +68,11 @@ All common IDE drives should work. Some drive geometries do not work, e.g. drives with hundreds of GiB of storage space, see [[!GNU_Savannah_bug 26425]]. -[[!toggle id="SATA" text="SATA drives may work in compatibility mode."]] -<!-- Sure? --[[tschwinge]] --> -[[!toggleable id="SATA" text=""" +## SATA + +SATA drives may work in compatibility mode. + This is how booting a [[GNU/Hurd_system|hurd]] will typically fail if GNU Mach couldn't connect to the hard disk, e.g., in a SATA system without IDE compatibility mode: @@ -81,7 +82,7 @@ compatibility mode: There *may* be an option in the system's BIOS setup to configure enabling such a compatibility mode. -"""]] + # Device Drivers diff --git a/microkernel/mach/gnumach/hardware_compatibility_list/discussion.mdwn b/microkernel/mach/gnumach/hardware_compatibility_list/discussion.mdwn index 69ca3190..2b65956a 100644 --- a/microkernel/mach/gnumach/hardware_compatibility_list/discussion.mdwn +++ b/microkernel/mach/gnumach/hardware_compatibility_list/discussion.mdwn @@ -1,4 +1,33 @@ +[[!meta copyright="Copyright © 2007, 2008, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + Further information may still be found on <http://www.nongnu.org/thug/gnumach_hardware.html> and could perhaps be incorporated into that page. --[[tschwinge]] + + +# SATA + +IRC, freenode, +hurd, 2011-07-24 + + <braunr> youpi: concerning the ide compatibility problem, it seems some + bioses provide several modes + <braunr> youpi: "legacy ide" and "native ide" + <braunr> i don't know what native ide really means, but when debugging ide + probing in gnumach, it just looks like there is nothing to detect + <braunr> and even in this mode, linux uses the ahci driver + <youpi> apparently native means it still uses the IDE protocol, but + possibly with other IRQs + <youpi> i.e. you need a PCI driver to handle that + <braunr> ok diff --git a/microkernel/mach/gnumach/memory_management.mdwn b/microkernel/mach/gnumach/memory_management.mdwn new file mode 100644 index 00000000..3e158b7c --- /dev/null +++ b/microkernel/mach/gnumach/memory_management.mdwn @@ -0,0 +1,135 @@ +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation open_issue_gnumach]] + +[[!toc]] + + +# IRC, freenode, #hurd, 2011-02-15 + + <braunr> etenil: originally, mach had its own virtual space (the kernel + space) + <braunr> etenil: in order to use linux 2.0 drivers, it now directly maps + physical memory, as linux does + <braunr> etenil: but there is nothing similar to kmap() or vmalloc() in + mach, so the kernel is limited to its 1 GiB + <braunr> (3 GiB userspace / 1 GiB kernelspace) + <braunr> that's the short version, there is a vmalloc() in mach, but this + trick made it behave almost like a kmalloc() + <antrik> braunr: the direct mapping is *only* for the benefit of Linux + drivers?... + <braunr> also, the configuration of segments limits the kernel space + <braunr> antrik: i'm not sure, as i said, this is the short version + <braunr> antrik: but there is a paper which describes the integration of + those drivers in mach + <etenil> you mean the linux 2.0 drivers? + <antrik> braunr: I read it once, but I don't remember anything about the + physical mapping in there... + <antrik> etenil: well, originally it was 1.3, but essentially that's the + same... + <braunr> i don't see any other reason why there would be a direct mapping + <braunr> except for performance (because you can use larger - even very + lage - pages without resetting the mmu often thanks to global pages, but + that didn't exist at the time) + + +# IRC, freenode, #hurd, 2011-02-15 + + <antrik> however, the kernel won't work in 64 bit mode without some changes + to physical memory management + <braunr> and mmu management + <braunr> (but maybe that's what you meant by physical memory) + + +## IRC, freenode, #hurd, 2011-02-16 + + <braunr> antrik: youpi added it for xen, yes + <braunr> antrik: but you're right, since mach uses a direct mapped kernel + space, the true problem is the lack of linux-like highmem support + <braunr> which isn't required if the kernel space is really virtual + + +# IRC, freenode, #hurd, 2011-06-09 + + <braunr> btw, how can gnumach use 1 GiB of RAM ? did you lower the + user/kernel boundary address ? + <youpi> I did + <braunr> 2G ? + <youpi> yes + <braunr> ok + <youpi> it doesn't make so much sense to let processes have 3G addressing + space when there can't be more that 1G physical memory + <braunr> that's sad for an operating system which does most things by + mapping memory eh + <youpi> well, if a process wants to map crazy things, 3G may be tight + already + <youpi> e.g. ext2fs + <braunr> yes + <youpi> so there's little point in supporting them + <braunr> we need hurd/amd64 + <youpi> and there's quite some benefit in shrinking them to 2G + <youpi> yes + <youpi> actually even 2G may become a bit tight + <youpi> webkit linking needs about 1.5-2GiB + <youpi> things become really crazy + <braunr> wow + <braunr> i remember the linux support for 4G/4G split when there was enough + RAM to fill the kernel space with struct page entries + + +# IRC, freenode, #hurd, 2011-11-12 + + <youpi> well, the Hurd doesn't "artificially" limits itself to 1.5GiB + memory + <youpi> i386 has only 4GiB addressing space + <youpi> we currently chose 2GiB for the kernel and 2GiB for the userspace + <youpi> since kernel needs some mappings, that leaves only 1.5GiB usable + physical memory + <sea4ever`> Hm? 2GiB for kernel, 2GiB for userspace, 500MiB are used for + what? + <youpi> for mappings + <youpi> such as device iomap + <youpi> contiguous buffer allocation + <youpi> and such things + <sea4ever`> Ah, ok. You map things in kernel space into user space then. + <youpi> linux does the same without the "bigmem" support + <youpi> no, just in kernel space + <youpi> kernel space is what determines how much physical memory you can + address + <youpi> unless using the linux-said-awful "bigmem" support + + +# IRC, freenode, #hurd, 2012-07-05 + + <braunr> hm i got an address space exhaustion while building eglibc :/ + <braunr> we really need the 3/1 split back with a 64-bits kernel + <pinotree> 3/1? + <braunr> 3 GiB userspace, 1 GiB kernel + <pinotree> ah + <braunr> the debian gnumach package is patched to use a 2/2 split + <braunr> and 2 GiB is really small for some needs + <braunr> on the bright side, the machine didn't crash + <braunr> there is issue with watch ./slabinfo which turned in a infinite + loop, but it didn't affect the stability of the system + <braunr> actually with a 64-bits kernel, we could use a 4/x split + + +# IRC, freenode, #hurd, 2012-08-10 + + <braunr> all modern systems embed the kernel in every address space + <braunr> which allows reduced overhead when making a system call + <braunr> sometimes there is no context switch at all + <braunr> on i386, there are security checks to upgrade the privilege level + (switch to ring 0), and when used, kernel page tables are global, so + they're not flushed + <braunr> using sysenter/sysexit makes it even faster + +[[open_issues/system_call_mechanism]]. diff --git a/microkernel/mach/gnumach/ports.mdwn b/microkernel/mach/gnumach/ports.mdwn index a29b8651..e7fdb446 100644 --- a/microkernel/mach/gnumach/ports.mdwn +++ b/microkernel/mach/gnumach/ports.mdwn @@ -1,15 +1,29 @@ -[[!meta copyright="Copyright © 2007, 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2007, 2008, 2009, 2011, 2012 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] * x86. This is the main port. + + * [[Xen]] + + * [[open_issues/64-bit_port]]. There is some preliminary work for a + x86\_64 port. + + * [[open_issues/ARM_port]]. Is not in a usable state. + * [PowerPC](http://www.pjbruin.dds.nl/hurd/). Is not in a usable state. - * Alpha. Was once started, but isn't in a usable state either. - * [[Xen]] + * Alpha: [project I](http://savannah.nongnu.org/projects/hurd-alpha), and + [project II](http://savannah.nongnu.org/projects/gnumach-alpha). Was once + started, but isn't in a usable state either. + + * MIPS. Status completely unknown. + + * [[open_issues/Mach_on_Top_of_POSIX]]. Status unknown. diff --git a/microkernel/mach/gnumach/ports/xen.mdwn b/microkernel/mach/gnumach/ports/xen.mdwn index c544fd57..c6023786 100644 --- a/microkernel/mach/gnumach/ports/xen.mdwn +++ b/microkernel/mach/gnumach/ports/xen.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2009 Free Software Foundation, +[[!meta copyright="Copyright © 2007, 2008, 2009, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -6,12 +6,13 @@ id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!toc]] -## Xen dom0, hypervisor + +# Xen dom0, hypervisor /!\ Now that GNU Mach handles PAE you can use a PAE-enabled hypervisor. @@ -20,7 +21,7 @@ You can either get binaries at <http://youpibouh.thefreecat.org/hurd-xen/> or bu - Copy `gnumach-xen-pae` and `hurd-modules` to your dom0 /boot. If you still have a non-PAE hypervisor, use `gnumach-xen-nonpae` instead. - Copy `hurd` into `/etc/xen`, edit it for fixing access to your hurd / and swap -## GNU/Hurd system +# GNU/Hurd system /!\ You need an already installed [[GNU/Hurd_system|hurd/running]]. @@ -30,7 +31,8 @@ If you have a free partition, you can fdisk to type 0x83, create a filesystem us Replace /dev/sda4 with your partition. Install and use crosshurd to setup a GNU/Hurd system on this partition. -## /etc/xen/hurd configuration + +# /etc/xen/hurd configuration Here is a sample /etc/xen/hurd configuration @@ -49,7 +51,8 @@ Suggestions about [[networking_configuration]] are available. If you need stable MAC addresses, use a syntax like `vif = [ 'mac=00:16:3e:XX:XX:XX, bridge=br0' ]`. -## Running Hurd with Xen + +# Running Hurd with Xen To run Hurd with Xen, use: @@ -63,7 +66,8 @@ and gnumach should get started. Proceed with native-install. - If `xm` complains about networking (`vif could not be connected`), it's Xen scripts' fault, see Xen documentation for how to configure the network. The simplest way is network-bridge with fixed IPs (note that you need the bridge-utils package for this). You can also just disable networking by commenting the vif line in the config. - If `xm` complains `Error: (2, 'Invalid kernel', 'xc_dom_compat_check: guest type xen-3.0-x86_32 not supported by xen kernel, sorry\n')`, you most probably have a PAE-enabled hypervisor and a non-PAE gnumach. Either install and boot non-PAE hypervisor and kernel, or rebuilt gnumach in PAE mode. -## Building from sources + +# Building from sources If you want to generate these images, first get the `gnumach-1-branch-Xen-branch` branch from gnumach CVS. Then look for "Ugly" in `kern/bootstrap.c`, how to generate `hurd-modules` is explained there, and you'll have to fix `EXT2FS_SIZE` and `LD_SO_SIZE` by hand. @@ -75,8 +79,49 @@ Then use The current `hurd-modules` was built from the debian packages `hurd 20070606-2` and `libc0.3 2.6.1-1`. /!\ This means that when using this image, your GNU/Hurd system also needs to be a glibc version 2.6 or later-based one! ---- +# `pv-grub` + +From Xen 4.0 on you can run the GNU Hurd directly using `pv-grub`, +without the need to [prepare a special bootstrap +image](http://youpibouh.thefreecat.org/hurd-xen/build_hurd-modules) (like an +initrd). + +Download http://youpibouh.thefreecat.org/hurd-xen/pv-grub.gz into /boot, and use the following for instance: + + kernel = "/boot/pv-grub.gz" + memory = 256 + disk = ['phy:sda4,hda,w'] + extra = "(hd0,1)/boot/grub/menu.lst" + vif = [ '' ] + +extra is now the path to the grub config file. + +# Partitions + +You will need the following notation for the gnumach root= parameter: + +root=part:2:device:hd0 + +to access the second partition of hd0, for instance. + +You will also need to use the parted storeio module for the /dev entries, for instance: + +settrans -fgap /dev/hd0s1 /hurd/storeio -T typed part:1:device:hd0 + +# Miscellaneous [[Internals]]. [[!GNU_Savannah_task 5468]], [[!GNU_Savannah_task 6584]]. + + +# Host-side Writeback Caching + +Optimization possible as it is with +[[QEMU|hurd/running/qemu/writeback_caching]]? + +IRC, freenode, #hurd, 2011-06-08 + + <braunr> youpi: does xen provide disk caching options ? + <youpi> through a blktap, probably + <braunr> ok diff --git a/microkernel/mach/gnumach/ports/xen/discussion.mdwn b/microkernel/mach/gnumach/ports/xen/discussion.mdwn new file mode 100644 index 00000000..2980e3b2 --- /dev/null +++ b/microkernel/mach/gnumach/ports/xen/discussion.mdwn @@ -0,0 +1,14 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + +Stuff from <http://youpibouh.thefreecat.org/hurd-xen> should be merged into +these pages here. diff --git a/microkernel/mach/gnumach/projects.mdwn b/microkernel/mach/gnumach/projects.mdwn index 47a2756c..f4ef192a 100644 --- a/microkernel/mach/gnumach/projects.mdwn +++ b/microkernel/mach/gnumach/projects.mdwn @@ -1,13 +1,13 @@ -[[!meta copyright="Copyright © 2005, 2006, 2007, 2008 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2005, 2006, 2007, 2008, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] This page is a place to keep track of ideas about things that may be improved in GNU Mach, so that it'll evolve to a reliable microkernel for The Hurd, both @@ -58,7 +58,8 @@ so that no duplicate efforts end up. * Improve the external pagers interface - * Implement read-ahead (huge I/O improvements expected). + * Implement [[open_issues/performance/io_system/read-ahead]] (huge I/O + improvements expected). * Making this interface synchronous should improve I/O performance significantly, without (almost) any drawbacks (we also get some diff --git a/microkernel/mach/gnumach/projects/clean_up_the_code.mdwn b/microkernel/mach/gnumach/projects/clean_up_the_code.mdwn index e865e61a..89a27b01 100644 --- a/microkernel/mach/gnumach/projects/clean_up_the_code.mdwn +++ b/microkernel/mach/gnumach/projects/clean_up_the_code.mdwn @@ -1,5 +1,5 @@ -[[!meta copyright="Copyright © 2005, 2006, 2007, 2008 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2005, 2006, 2007, 2008, 2010, 2012 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -9,6 +9,8 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] +[[!tag open_issue_gnumach]] + # Restructure the tree in a sane way Merge `linux/src` and `linux/dev`. But only if using a sane RCS, so leave it @@ -119,3 +121,12 @@ further files (also exported ones) that serve no real value, but are being # Rewrite ugly code + + +# IRC, freenode, #hurd, 2012-09-06 + + <mcsim> hello. Why size parameter of rpc device_read has type + "mach_msg_type_number_t *"? Why not just "vm_size_t *"? + <mcsim> this parameter has name data_count + <braunr> that's one of the reasons mach is confusing + <braunr> i can't really tell you why, it's messy :/ diff --git a/microkernel/mach/gnumach/projects/gdb_stubs.mdwn b/microkernel/mach/gnumach/projects/gdb_stubs.mdwn index ef1b4909..064da7bf 100644 --- a/microkernel/mach/gnumach/projects/gdb_stubs.mdwn +++ b/microkernel/mach/gnumach/projects/gdb_stubs.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2010 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -8,6 +8,12 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] +[[!tag open_issue_gnumach open_issue_gdb]] + * <http://lists.gnu.org/archive/html/bug-hurd/2008-04/msg00103.html> * [ChangeLog.gdb](http://cvs.savannah.gnu.org/viewvc/gnumach/ChangeLog.gdb?root=hurd&view=markup&pathrev=gnumach-1-branch-gdb-branch) + +This may be another follow-up project: [*Linux Kernel GDB tracepoint +module*](http://thread.gmane.org/gmane.comp.gdb.devel/29369), Hui Zhu, +2010-10-09. diff --git a/microkernel/mach/history.mdwn b/microkernel/mach/history.mdwn index a8951737..776bb1d7 100644 --- a/microkernel/mach/history.mdwn +++ b/microkernel/mach/history.mdwn @@ -1,7 +1,3 @@ -# <a name="Table_of_Contents"> Table of Contents </a> - -%TOC% - # <a name="Early_beginnings"> Early beginnings </a> Mach has quite a history. Everything actually started at the University of Rochester in 1975. It was invented to demonstrate how operating systems could be built using a modular design where processes communicated using message passing, even across networks. The system was called the Rochester Intelligent Gateway and ran on a 16 bit mini computer called Eclipse from Data General. @@ -62,3 +58,23 @@ Verbatim copying and distribution of this entire article is permitted in any med Apple's Macintosh OSX (OS 10.x) is based on [Darwin](http://www.apple.com/macosx/technologies/darwin.html). _"Darwin uses a monolithic kernel based on [[TWiki/FreeBSD]] 4.4 and the OSF/mk Mach 3."_ Darwin also has a [Kernel Programming](http://developer.apple.com/techpubs/macosx/Darwin/General/KernelProgramming/About/index.html) Book. -- [[Main/GrantBow]] - 22 Oct 2002 + +IRC, freenode, #hurd, 2012-08-29: + + <pavlx> was moved the page from apple.com about darwin kernel programming + as described on the + https://www.gnu.org/software/hurd/microkernel/mach/history.html + <pavlx> i found the page and it's + https://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/KernelProgramming/About/About.html + <pavlx> it's not anymore the old page + http://developer.apple.com/techpubs/macosx/Darwin/General/KernelProgramming/About/index.html + <pavlx> and the link about darwin does noit exists anymore ! the new one + could be https://ssl.apple.com/science/profiles/cornell + <pavlx> the old one was + http://www.apple.com/macosx/technologies/darwin.html + <pavlx> the link to Darwin is changed i suppose that the nw one it's + https://ssl.apple.com/science/profiles/cornell + <pavlx> and the link to Kern Programming it's + https://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/KernelProgramming/About/About.html + <pavlx> can't be anymore + http://developer.apple.com/techpubs/macosx/Darwin/General/KernelProgramming/About/index.html diff --git a/microkernel/mach/ipc.mdwn b/microkernel/mach/ipc.mdwn index aaf3ba23..1bb44b59 100644 --- a/microkernel/mach/ipc.mdwn +++ b/microkernel/mach/ipc.mdwn @@ -1,22 +1,21 @@ -[[!meta copyright="Copyright © 2007, 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2007, 2008, 2010 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -[[General_information|/ipc]] about IPC. +Read about the [[general concept of *inter-process communication* (IPC)|/ipc]]. -An IPC is sent by invoking a [[port]]. <!-- Isn't this wording a bit strange? -``IPC is sent'' --tschwinge --> +On Mach, an IPC is done by invoking a [[port]]. + +The two fundamental operations, to *send* and *receive* [[message]]s, are used +to implement a [[RPC]] system. [[Sequence_numbering]]. [The Unofficial GNU Mach IPC beginner's guide](http://www.nongnu.org/hurdextras/ipc_guide/ipc_guide.html) - -# See Also - -* [[RPC]] diff --git a/microkernel/mach/memory_object.mdwn b/microkernel/mach/memory_object.mdwn new file mode 100644 index 00000000..f32fe778 --- /dev/null +++ b/microkernel/mach/memory_object.mdwn @@ -0,0 +1,33 @@ +[[!meta copyright="Copyright © 2002, 2003, 2010, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +Mach's [[virtual_memory]] subsystem uses *memory objects* for supplying the +content of regions of virtual memory in an [[virtual_address_space]]. + +All of these objects are managed by *memory manager*s, that are also called +*pager*s. These can be implemented as user-space processes. + +Both the memory objects, and their managers are kernel objects, and are +accessed by [[port]]s. + +A system's physical memory is conceived as a *memory cache* that contains +*memory cache objects*. So when a [[thread]] accesses a page in its task's +address space, the memory object that includes this page is *cached* in the +memory cache. Memory objects are [[paged out and paged +in|external_pager_mechanism]] by the aforementioned memory managers. The +decision when they should be paged in or paged out is left to [[Mach]]. Each +memory object has an ordered list of memory managers that provide paging. The +last one tried is the *default memory manager* that resides in the microkernel, +in contrast to most of the others. The default memory manager is needed +because the microkernel can't wait infinitely for someone else to free the +memory cache: it just calls the next memory manager hoping it to succeed. + +Read about [[GNU Mach's memory management|gnumach/memory_management]]. diff --git a/microkernel/mach/memory_object/discussion.mdwn b/microkernel/mach/memory_object/discussion.mdwn new file mode 100644 index 00000000..907f859a --- /dev/null +++ b/microkernel/mach/memory_object/discussion.mdwn @@ -0,0 +1,74 @@ +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation open_issue_gnumach]] + +[[!toc]] + + +# IRC, freenode, #hurd, 2011-08-05 + + < neal> braunr: For instance, memory objects are great as they allow you to + specify the mapping policy in user space. + < neal> braunr: But, the policy for determining the eviction order is + realized by Mach + < neal> braunr: And user-space has no control + < braunr> are you referring to the page cache lru approximation and stuff + like resource containers ? + < neal> I'm not sure what you mean by page cache lru appoximateion + < braunr> the kernel eviction policy :) + < neal> that's an implementation detail + + +# IRC, freenode, #hurd, 2011-09-05 + + <braunr> mach isn't a true modern microkernel, it handles a lot of + resources, such as high level virtual memory and cpu time + <braunr> for example, the page replacement mechanism can't be implemented + outside the kernel + <braunr> yet, it provides nothing to userspace server to easily allocate + resources on behalf of clients + <braunr> so, when a thread calls an RPC, the cpu time used to run that RPC + is accounted on the server task + <braunr> the hurd uses lots of external memory managers + +[[external_pager_mechanism]]. + + <braunr> but they can't decide how to interact with the page cache + <braunr> the kernel handles the page cache, and initiates the requests to + the pagers + <cjuner> braunr, why can't they decide that? + <braunr> because it's implemented in the kernel + <braunr> and there is nothing provided by mach to do that some other way + <slpz_> braunr: you probably already know this, but the problem with client + requests being accounted on behalf the server, is fixed in Mach with + Migrating Threads + +[[open_issues/mach_migrating_threads]]. + + <braunr> slpz_: migrating threads only fix the issue for the resources + managed by mach, not the external servers + <braunr> slpz_: but it's a (imo necessary) step to completely solve the + issue + <braunr> in addition to being a great feature for performance (lighter + context switchers, less state to track) + <braunr> it also helps priority inversion problems + <slpz_> braunr: I was referring just to cpu-time, but I agree with you an + interface change is needed for external pagers + <braunr> slpz_: servers in general, not necessarily pagers + <slpz_> as a way to mitigate the effect of Mach paging out to external + pagers, the folks at OSF implemented an "advisory pageout", so servers + are "warned" that they should start paging out, and can decide which + pages are going to be flushed by themselves + +[[open_issues/resource_management_problems]]. + + +# [[open_issues/memory_object_model_vs_block-level_cache]] diff --git a/microkernel/mach/message.mdwn b/microkernel/mach/message.mdwn new file mode 100644 index 00000000..4c49af17 --- /dev/null +++ b/microkernel/mach/message.mdwn @@ -0,0 +1,33 @@ +[[!meta copyright="Copyright © 2002, 2003, 2010, 2012 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +*Messages* are collections of typed data, with a defined layout, including an +[[ID|msgh_id]]. + +They are used for [[IPC]], and are sent to and received from [[port]]s using +the `mach_msg` interface. + +These messages are not only opaque data. They can also contain [[port +rights|port]] to be passed to another [[task]]. Port rights are either +*copied* or *moved*. Notice that port receive right must be moved but not +copied because there can't be more than one task that holds the receive right +to a port. The receiving task creates new local port name to the port rights +it received. + +Some data in the message can be *out-of-line data*. In the message, these are +*references* to memory regions ([[memory_object]]s) that are *virtually +copied*. When the message is received in a task, these virtual copies become +part of the task by mapping them into the receiver's [[virtual_address_space]]. +Another key concept that is applied is using *copy-on-write*, which means that +data is not copied immediately, but only when it is changed. This is primarily +used to send large blocks of data efficiently, as it is too expensive to store +them in the kernel address space: extra copied need only be made at the moment +that the memory regions begin to diverge, by threads modifying them. diff --git a/microkernel/mach/message/msgh_id.mdwn b/microkernel/mach/message/msgh_id.mdwn new file mode 100644 index 00000000..986fcbc7 --- /dev/null +++ b/microkernel/mach/message/msgh_id.mdwn @@ -0,0 +1,254 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_mig]] + +Every [[message]] has an ID field, which is defined in the [[RPC]] `*.defs` +files. + + +# IRC, freenode, #hurd, 2012-07-12 + +[Extending an existing RPC.] + + <antrik> create a new call, either with a new variant of vm_statistics_t, + or a new structure with only the extra fields + <braunr> that seems cleaner indeed + <braunr> but using different names for the same thing seems so tedious and + unnecessary :/ + <antrik> it's extra effort, but it pays off + <braunr> i agree, it's the right way to do it + <braunr> but this implies some kind of versioning + <braunr> which is currently more or less done using mig subsystem numbers, + and skipping obsolete calls in rpc definition files + <braunr> and a subsystem is like 100 calls (200 with the replies) + <braunr> at some point we should recycle them + <braunr> or use truely huge ranges + <antrik> braunr: that's not something we need to worry about until we get + there -- which is not likely to happen any time soon :-) + <braunr> "There is no more room in this interface for additional calls." + <braunr> in mach.defs + <braunr> i'll use the mach4.defs file + <braunr> but it really makes no sense at all to do such things just because + we want to be compatible with 20 year old software nobody uses any more + <braunr> who cares about the skips used to keep us from using the old mach + 2.5 interface .. + <braunr> (and this 100 arbitrary limit is really ugly too) + <antrik> braunr: I agree that we don't want to be compatible with 20 years + old software. just Hurd stuff from the last few years is perfectly fine. + <tschwinge> braunr, antrik: I agree with the approach of using a new + RPC/data structure for incompatible changes, and I also agree that + recycling RPC slots that have been unused (skipped) for some years is + fine. + <antrik> tschwinge: well, we probably shouldn't just reuse them + arbitrarily; but rather do a mass purge if the need really arises... + <antrik> it would be confusing otherwise IMHO + <tschwinge> antrik: What do you understand by doing a mass purge? + <tschwinge> My idea indeed was to replace arbitrary "skip"s by new RPC + definitions. + <braunr> a purge would be good along with a mig change to make subsystem + and routines identifier larger + <braunr> i guess 16-bits width should do + <tschwinge> But what do you unterstand by a "purge" in this context. + <braunr> removing all the skips + <tschwinge> But that moves the RPC ids following after? + <braunr> yes + <braunr> that's why i think it's not a good thing, unless we also change + the numbering + <tschwinge> ... which is a incompatible change for all clients. + <braunr> yes + <tschwinge> OK, so you'd propose a new system and deprecate the current + one. + <braunr> not really new + <braunr> just larger numbers + <braunr> we must acknowledge interfaces change with time + <tschwinge> Yes, that's "new" enough. ;-) + <tschwinge> New in the sense that all clients use new iterfaces. + <braunr> that's enough to completely break compability, yes + <braunr> at least binary + <tschwinge> Yes. + <tschwinge> However, I don't see an urgent need for that, do you? + <tschwinge> Why not just recycled a skip that has been unused for a decade? + <braunr> i don't think we should care much about that, as the only real + issue i can see is when upgrading a system + <braunr> i don't say we shouldn't do that + <braunr> actually, my current patch does exactly this + <tschwinge> OK. :-) + <braunr> purging is another topic + <braunr> but purging without making numbers larger seems a bit pointless + <braunr> as the point is allowing developers to change interfaces without + breaking short time compability + <braunr> compatibility* + <braunr> also, interfaces, even stable, can have more than 100 calls + <braunr> (at the same time, i don't think there would ever be many + interfaces, so using 16-bits integers for the subsystems and the calls + should really be fine, and cleanly aligned in memory) + <antrik> tschwinge: you are right, it was a brain fart :-) + <antrik> no purge obviously + <antrik> but I think we only should start with filling skips once all IDs + in the subsystem are exhausted + <antrik> braunr: the 100 is not fixed in MIG IIRC; it's a definition we + make somewhere + <antrik> BTW, using multiple subsystems for "overflowing" interfaces is a + bit ugly, but not to bad I'd say... so I wouldn't really consider this a + major problem + <antrik> err... not too bad + <antrik> especially since Hurd subsystem usually are spaced 1000 aways, so + there are some "spare" blocks between them anyways + <braunr> hm i'm almost sure it's related to mig + <braunr> that's how the reply id is computed + <antrik> of course it is related to MIG... but I have a vague recollection + that this constant is not fixed in the MIG code, but rather supplied + somewhere. might be wrong though :-) + <pinotree> you mean like the 101-200 skip block in hurd/tioctl.defs? + <antrik> pinotree: exactly + <antrik> these are reserved for reply message IDs + <antrik> at 200 a new request message block begins... + <braunr> server.c: fprintf(file, "\tOutP->Head.msgh_id = InP->msgh_id + + 100;\n"); + <braunr> it's not even a define in the mig code :/ + <pinotree> meaning that in the space of an hurd subsystem there are max 500 + effective rpc's? + <antrik> actually, ioctls are rather special, as the numbers are computed + from the ioctl properties... + <antrik> braunr: :-( + <braunr> pinotree: how do you get this value ? + <pinotree> braunr: 1000/2? :) + <braunr> ? + <braunr> why not 20000/3 ? + <antrik> pinotree: yes + <braunr> where do they come from ? + <braunr> ah ok sorry + <pinotree> braunr: 1000 is the space of each subsystem, and each rpc takes + an id + its replu + <pinotree> *reply + <braunr> right + <braunr> 500 is fine + <braunr> better than 100 + <braunr> but still, 64k is way better + <braunr> and not harder to do + <pinotree> (hey, i'm the noob in this :) ) + <antrik> braunr: it's just how "we" lay out subsystems... nothing fixed + about it really; we could just as well define new subsystems with 10000 + or whatever if we wanted + <braunr> yes + <braunr> but we still have to consider this mig limit + <antrik> there are one or two odd exceptions though, with "related" + subsystems starting at ??500... + <antrik> braunr: right. it's not pretty -- but I wouldn't consider it + enough of a problem to invest major effort in changing this... + <braunr> agreed + <braunr> at least not while our interfaces don't change often + <braunr> which shouldn't happen any time soon + + <tschwinge> Hmm, I also remember seeing some emails about indeed versioning + RPCs (by Roland, I think). I can try to look that up if there's + interest. + + <braunr> i'm only adding a cached pages count you know :) + <braunr> (well actually, this is now a vm_stats call that can replace + vm_statistics, and uses flavors similar to task_info) + <antrik> braunr: I don't think introducing "flavors" is a good idea + <braunr> i just did it the way others calls were done + <braunr> other* + <braunr> woud you prefer a larger structure with append-only upgrades ? + <antrik> I prefer introducing new calls. it avoids an unncessary layer of + indirection + <antrik> flavors are not exactly RPC-over-RPC, but definitely going down + that road... + <braunr> right + <antrik> as fetching VM statistics is not performance-critical, I would + suggest adding a new call with only the extra stats you are + introducing. then if someone runs an old kernel not implementing that + call, the values are simply left blank in the caller. makes + backward-compatibility a no-brainer + <antrik> (the alternative is a new call fetching both the traditional and + the new stats -- but this is not necessary here, as an extra call + shouldn't hurt) + <braunr> antrik: all right + + +## IRC, freenode, #hurd, 2012-07-13 + + <braunr> so, should i replace old, unused mach.defs RPCs with mine, or add + them to e.g. mach4.defs ? + <antrik> braunr: hm... actually I wonder whether we shouldn't add a + gnumach.defs -- after all, it's neither old mach nor mach4 interfaces... + <braunr> true + <braunr> good idea + <braunr> i'll do just that + <braunr> hm, doesn't adding a new interface file requires some handling in + glibc ? + <youpi> simply rebuild it + <braunr> youpi: no i mean + <braunr> youpi: glibc knows about mach.defs and mach4.defs, but i guess we + should add something so that it knows about gnumach.defs + <youpi> ah + <youpi> probably, yes + <braunr> ok + <braunr> i don't understand why these files are part of the glibc headers + <pinotree> are they? + <braunr> (i mean mach_interface.h and mach4.h) + <braunr> for example + <braunr> youpi: the interface i'll add is vm_cache_statistics(task, + &cached_objects, &cached_pages) + <braunr> if it's ok i'll commit directly into the gnumach repository + <youpi> shouldn't it rather be a int array, to make it extensible? + <youpi> like other stat functions of gnumach + <braunr> antrik was against doing that + <braunr> well, he was against using flavors + <braunr> maybe we could have an extensible array yes, and require additions + at the end of the structure + + +## IRC, freenode, #hurd, 2012-07-14 + + <antrik> braunr: there are two reasons why the files are part of glibc. one + is that glibc itself uses them, so it would be painful to handle + otherwise. the other is that libc is traditionally responsible for + providing the system interface... + <antrik> having said that, I'm not sure we should stick with that :-) + <braunr> antrik: what do you think about having a larger structure with + reserved fields ? sounds a lot better than flavors, doesn't it ? + <youpi> antrik: it's in debian, yes + <braunr> grmbl, adding a new interface just for a single call is really + tedious + <braunr> i'll just add it to mach4 + <antrik> braunr: well, it's not unlikely there will be other new calls in + the future... but I guess using mach4.defs isn't too bad + <antrik> braunr: as for reserved fields, I guess that is somewhat better + than flavors; but I can't say I exactly like the idea either... + <braunr> antrik: there is room in mach4 ;p + + +## IRC, freenode, #hurd, 2012-07-23 + + <tschwinge> I'm not sure yet whether I'm happy with adding the RPC to + mach4.defs. + <braunr> that's the only question yes + <braunr> (well, no, not only) + <braunr> as i know have a better view of what's involved, it may make sense + to create a gnumach.defs file + <braunr> tschwinge: all right i'll create a gnumach.defs file + <tschwinge> braunr: Well, if there is general agreement that this is the + way to go. + <tschwinge> braunr: In that case, I guess there's no point in being more + fine-grained -- gnumach-vm.defs or similar -- that'd probably be + over-engineering. If the glibc bits for libmachuser are not + straight-forward, I can help with that of course. + <braunr> ok + + +## IRC, freenode, #hurd, 2012-07-27 + + <braunr> tschwinge: i've pushed a patch on the gnumach page_cache branch + that adds a gnumach.defs interface + <braunr> tschwinge: if you think it's ok, i'll rewrite a formal changelog + so it can be applied diff --git a/microkernel/mach/mig.mdwn b/microkernel/mach/mig.mdwn index 4275a4b4..331b3bf4 100644 --- a/microkernel/mach/mig.mdwn +++ b/microkernel/mach/mig.mdwn @@ -1,21 +1,34 @@ -[[!meta copyright="Copyright © 2001, 2002, 2003, 2006, 2007, 2008 Free Software -Foundation, Inc."]] +[[!meta copyright="Copyright © 2001, 2002, 2003, 2006, 2007, 2008, 2010 Free +Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -The Mach Interface Generator (MIG) is an [[IDL]] compiler. Based on an -interface definition, it creates stubs to [[invoke]] object methods -and to demultiplex incoming messages. These stubs conveniently hide -the details of Mach's [[IPC]] machinery and make it easy to implement -and use Mach [[interface]]s as [[remote_procedure_calls_(RPC)|rpc]]. +The *Mach Interface Generator* (*MIG*) is an [[IDL]] compiler. Based on an +interface definition, it creates stub code to [[invoke]] object methods and to +demultiplex incoming messages. These stub functions conveniently hide the +details of Mach's [[IPC]] and [[port]] machinery and make it easy to implement +and use Mach [[interface]]s as [[remote procedure calls (RPC)|rpc]]: by using +the stub functions, the client programs can call remote procedures more or less +like any other C function. + +These functions encode arguments into [[message]]s' format (*marshalling*), +wait for a result on a newly created [[reply port|port]], decode return +arguments from the reply message (*demarshalling*, or *unmarshalling*) and pass +them to the client program. Similar actions are provided in the skeletons that +are linked to server programs. + +MIG allows very precise semantics to be specified about what the arguments are +and how to be passed. + + + * [[Documentation]] -* [[Documentation]] # Implementations diff --git a/microkernel/mach/mig/documentation.mdwn b/microkernel/mach/mig/documentation.mdwn index 82d51a72..7d4f1eca 100644 --- a/microkernel/mach/mig/documentation.mdwn +++ b/microkernel/mach/mig/documentation.mdwn @@ -1,13 +1,13 @@ -[[!meta copyright="Copyright © 2002, 2003, 2005, 2007, 2008, 2009 Free Software -Foundation, Inc."]] +[[!meta copyright="Copyright © 2002, 2003, 2005, 2007, 2008, 2009, 2010 Free +Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] This is a small collection of links to external documents describing the *Mach Interface Generator* used by GNU Mach. @@ -17,23 +17,23 @@ Interface Generator* used by GNU Mach. A tutorial which demonstrates the use of the C Threads library primitives in writing a multithreaded program and the use of the Mach Interface Generator -(MIG) to generate remote procedure calls for interprocess communication. Like +(MIG) to generate remote procedure calls for inter-process communication. Like its companion tutorial, it is based on the Mach 2.5 system. However, the concepts are applicable to Mach 3.0 user level programming. Linda R. Walmer and Mary R. Thompson. *A Programmer's Guide to the Mach User Environment*. [PostScript -](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/machuse.ps), -[Doc](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/machuse.doc). +](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/machuse.ps), +[Doc](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/machuse.doc). February 1988. School of Computer Science, Carnegie Mellon University. An ftp directory containing the [mig programming -examples](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig_example) +examples](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig_example) for this tutorial. Slides to Rich Drave's talk on MIG, on November 21, 1991: -[PostScript](ftp://ftp.cs.cmu.edu//afs/cs/project/mach/public/doc/unpublished/internals_slides/Mig/root.ps), -[TeX](ftp://ftp.cs.cmu.edu//afs/cs/project/mach/public/doc/unpublished/internals_slides/Mig/slides.tex). +[PostScript](http://www.cs.cmu.edu//afs/cs/project/mach/public/doc/unpublished/internals_slides/Mig/root.ps), +[TeX](http://www.cs.cmu.edu//afs/cs/project/mach/public/doc/unpublished/internals_slides/Mig/slides.tex). # Roots @@ -41,15 +41,15 @@ Slides to Rich Drave's talk on MIG, on November 21, 1991: Mig is an implementation of a subset of the Matchmaker **language**. "Matchmaker is a language for specifying and automating the generation of -multilingual interprocess communication interfaces. MIG is an interim +multilingual inter-process communication interfaces. MIG is an interim implementation of a subset of the Matchmaker language that generates C and C++ -remote procedure call interfaces for interprocess communication between Mach +remote procedure call interfaces for inter-process communication between Mach tasks." Richard P. Draves, Michael B. Jones, Mary R. Thompson, *MIG - THE MACH INTERFACE GENERATOR*. -[ps](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig.ps), -[doc](ftp://ftp.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig.doc). +[ps](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig.ps), +[doc](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig.doc). November 1989. Department of Computer Science, Carnegie-Mellon University. @@ -70,6 +70,12 @@ pp. 67--77." [Chapter 4, Inter Process Communication](http://www.gnu.org/software/hurd/gnumach-doc/Inter-Process-Communication.html). + * OSF's [Server Writer's Guide (ps)](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/server_writer.ps) + [Server Writer's Guide (pdf)](http://shakthimaan.com/downloads/hurd/server_writer.pdf) + + * OSF's [Server Writer's Interfaces (ps)](http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/osf/server_interface.ps) + [Server Writer's Interfaces (pdf)](http://shakthimaan.com/downloads/hurd/server_interface.pdf) + * Flags: * [[dealloc_and_dealloc[]|dealloc]] diff --git a/microkernel/mach/mig/gnu_mig.mdwn b/microkernel/mach/mig/gnu_mig.mdwn index 1bcbd545..0de1bd67 100644 --- a/microkernel/mach/mig/gnu_mig.mdwn +++ b/microkernel/mach/mig/gnu_mig.mdwn @@ -1,13 +1,13 @@ -[[!meta copyright="Copyright © 2001, 2006, 2008, 2009 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2001, 2006, 2008, 2009, 2010 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] GNU MIG is the GNU distribution of the [[Mach_3.0_interface_generator_*MIG*|mig]], as maintained by the GNU Hurd @@ -20,5 +20,9 @@ software in the GNU system that uses Mach-based GNU MIG is fully compatible with [[OSF_MIG|mig]]. +Like its predecessor, it can only generate C code, that has to be compiled and +linked to client and server programs respectively ([[!taglink +open_issue_mig]]). + * [[Building]] - building (and obtaining) GNU MIG * [[Open Issues|tag/open_issue_mig]] diff --git a/microkernel/mach/mig/gnu_mig/building.mdwn b/microkernel/mach/mig/gnu_mig/building.mdwn index f92f7dbe..e7d3c150 100644 --- a/microkernel/mach/mig/gnu_mig/building.mdwn +++ b/microkernel/mach/mig/gnu_mig/building.mdwn @@ -1,15 +1,28 @@ +[[!meta copyright="Copyright © 2006, 2007, 2008, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + # <a name="Building_the_Mach_Interface_Gene"> Building the Mach Interface Generator from Source </a> -If you want to build the Mach Interface Generator yourself instead of just using a pre-built package, follow these instructions. +If you want to build the Mach Interface Generator yourself instead of just +using a pre-built package, follow these instructions. ## <a name="Getting_the_Source_Code"> Getting the Source Code </a> You can chose between getting the [sources from the developers' -RCS](http://savannah.gnu.org/cvs/?group=hurd): +RCS](http://git.savannah.gnu.org/cgit/hurd/): - $ cvs -z3 -d:pserver:anonymous@cvs.savannah.gnu.org:/sources/hurd co mig + $ git clone http://git.savannah.gnu.org/cgit/hurd/mig.git/ -... or (if you are working on a Debian system) the ones that are used for the [current Debian mig package](http://packages.debian.net/source/unstable/mig): +... or (if you are working on a Debian system) get the sources that are used for the +[current Debian mig package](http://packages.debian.net/source/unstable/mig): $ apt-get source mig @@ -17,53 +30,70 @@ Please see the Debian [[hurd/running/debian/FAQ]] before using _apt-get source_. The unpacked source tree is around 1 MiB, and the build tree also is around 1 MiB. -## <a name="Preparing_for_the_Build"> Preparing for the Build </a> +## <a name="_on_Debian_systems"> On Debian Systems: </a> -### <a name="_on_Debian_systems"> ... on Debian systems </a> +### <a name="Preparing_for_the_Build"> Preparing for the Build </a> -Building the Mach Interface Generator requires the _build-essential_ and _fakeroot_ packages, their dependencies and additional packages that are specified by the source mig package: +Building MIG requires the *build-essential* and *fakeroot* packages, +and some additional dependencies specified by the mig source package: # apt-get install build-essential fakeroot # apt-get build-dep mig -### <a name="_on_non_Debian_systems"> ... on non-Debian systems </a> +### <a name="Building_and_Installing"> Building and Installing </a> <a name="_a_deb_file"> ... a _.deb_ file </a> -Building the Mach Interface Generator requires a C compiler, a standard C library (with corresponding header files) and your favourite flavor of awk (gawk), yacc (bison), lex (flex) and make. +Change into the directory with the downloaded / unpacked MIG sources: -Additionally, you need to have GNU Mach's header files installed. See -[[mach/gnumach/building]] about how to do that, then come back here. + $ cd mig-X.X.X.XX -## <a name="Building_and_Installing"> Building and Installing </a> +Start the build process: -### <a name="_a_deb_file"> ... a _.deb_ file </a> + $ dpkg-buildpackage -us -uc -b -rfakeroot -Change into the directory with the downloaded / unpacked MIG sources (_mig-1.3.1.99_): +This will create a _.deb_ package in the parent directory, +which you can then install on your system. - $ cd mig-1.3.1.99 +## <a name="_on_non_Debian_systems"> On non-Debian Systems: </a> -Start the build process: +### <a name="Preparing_for_the_Build"> Preparing for the Build </a> - $ dpkg-buildpackage -us -uc -b -rfakeroot +Building the Mach Interface Generator requires a C compiler, a standard 32 bit +C library (with corresponding header files), your favourite flavor of awk +(gawk), yacc (bison), lex (flex) and make. -You can then install / distribute the _.deb_ file which will drop out one directory above the current one. +Additionally, you need to have GNU Mach's header files installed. See +[[building GNU Mach|mach/gnumach/building]] about how to do that, then come back here. + +### <a name="Building_and_Installing"> Building and Installing </a> + +First, generate the configuration files: -### <a name="_TODO_"> [TODO] </a> + $ cd mig + $ autoreconf --install -The Mach Interface Generator has to be built in a separate directory: +The Mach Interface Generator has to be built in a separate build directory: - $ mkdir mig-build - $ cd mig-build + $ mkdir build + $ cd build + +Find the base directory where you installed GNU Mach's header files and where +you now intend to install the Mach Interface Generator (e.g. _~/gnu_), and run +configure: + + $ GNU=~/gnu + $ TARGET_CPPFLAGS=-I"$GNU"/include ../configure --prefix="$GNU" -Find the root directory where you installed GNU Mach's header files and where you now intend to install the Mach Interface Generator (_~/gnu_) and the path to your Mach Interface Generator sources (\_[...]/mig) and configure it: +If you are building on a 64 bit machine, you need to add a --host option: $ GNU=~/gnu - $ TARGET_CPPFLAGS=-I"$GNU"/include [...]/mig/configure --prefix="$GNU" + $ TARGET_CPPFLAGS=-I"$GNU"/include ../configure --prefix="$GNU" --host=i686-unknown-linux-gnu -Build and install the Mach Interface Generator into _$GNU_, i.e. _~/gnu/_ in our example: +Build and install the Mach Interface Generator into _$GNU_ (i.e. _~/gnu/_ in our example): $ make all install -To make your _mig_ binary easily available, you should append something like the following to e.g. your _~/.bash\_profile_: +To make your _mig_ binary easily available, you should append something like +the following to e.g. your _~/.bash\_profile_: PATH=~/gnu/bin:$PATH export PATH diff --git a/microkernel/mach/mig/gnu_mig/building/discussion.mdwn b/microkernel/mach/mig/gnu_mig/building/discussion.mdwn new file mode 100644 index 00000000..d7636158 --- /dev/null +++ b/microkernel/mach/mig/gnu_mig/building/discussion.mdwn @@ -0,0 +1,16 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +# Non-cross-compiling + +[[!tag open_issue_mig]] + +[[samuelthibault]] mentioned that I should make clear what compiler options, etc. are only needed if compiling on a 64 bit computer. However, I don't know if the --host=i686... option is needed, here and when making gnumach, in case there may be some other default on 32 bit computers? --[[sudoman]] + diff --git a/microkernel/mach/pmap.mdwn b/microkernel/mach/pmap.mdwn new file mode 100644 index 00000000..6910bfd3 --- /dev/null +++ b/microkernel/mach/pmap.mdwn @@ -0,0 +1,74 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation open_issue_gnumach]] + + +# IRC, freenode, #hurd, 2012-02-01 + + <sekon> on Hurd what is the difference between kernel memory object and + pmap module ?? + <sekon> pmap is heap/libraries table for each thread while kernel memory + object refers to arbitary blobs of data ?? + <braunr> sekon: pmap is the low level memory mapping module + <braunr> i.e. it programs the mmu + <braunr> and these aren't hurd-specific, they are mach modules + <sekon> braunr: so kernel memonry objects consists of a bunch of pmaps ?? + <braunr> sekon: memory objects can be various things, be specific please + <braunr> (they're certainly not a bunch of pmaps though, no) + <braunr> there is one pmap per vm_map, and there is one vm_map per task + <braunr> and there is no need for double question marks, is ther ?? + <sekon> lol then is kernel memory object , please excuse the metaphor + something like a base class for pmap + <braunr> i don't know what a "kernel memory object" is, be specific please, + again + <sekon> braunr: + http://courses.cs.vt.edu/~cs5204/fall05-gback/presentations/MachOS_Rajesh.ppt + <sekon> goto page titled External Memory Management (EMM) on page 15 + <sekon> Kernel memory object shows up + <braunr> you know there are other formats for this document + <sekon> nope .. i did not know that + <sekon> in page 17 pmamp shows up + <braunr> "the problems of external memory management" ? + <sekon> braunr: the paper i am also reading is called x15mach_thesis + <braunr> ah, that's mine + * sekon bows + <sekon> :) + <braunr> ok i see page 17 + <sekon> so please good sir explain the relationship between kernel memory + object and pmap + <sekon> (if any) + <sekon> braunr: there is no mention of kernel memory object + <braunr> again, i don't see any reference or definition of "kernel memory + object" + <sekon> but your paper says + <sekon> that when page faults occur + <sekon> the kernel contact the manager for a kernel reference object + <sekon> *memory + <braunr> where ? + <sekon> in section 2.1.3 (unless i read it wrong) + <sekon> no just a sec + <sekon> 2.1.5 + <braunr> i never used the expression "kernel memory object" there :p + <braunr> anyway, you're referring simple to memory objects as seen by + userspace pagers + <braunr> a memory object is a data container + <braunr> usually, it's a file + <braunr> but it can be anything + <braunr> the pager is the task that provides its content and implements the + object methods + <braunr> as for the relation between them and the pmap module, it's a + distant one + <braunr> i'll explain it with an example + <braunr> page fault -> request content of memory object at a given offset + with given length from pager -> ask pmap to establish the mapping in the + mmu + <sekon> braunr: thank you ver much + <sekon> *very diff --git a/microkernel/mach/port.mdwn b/microkernel/mach/port.mdwn index af4a0c8d..ccc7286f 100644 --- a/microkernel/mach/port.mdwn +++ b/microkernel/mach/port.mdwn @@ -1,41 +1,89 @@ -[[!meta copyright="Copyright © 2007, 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2002, 2003, 2007, 2008, 2010, 2011, 2012 Free +Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] - -Mach ports are [[capabilities|capability]]. - -A Mach port is a kernel queue. Each port has associated with -it a receive right and one or more send and send-once rights. -A queue can hold a number of messages. Once the queue is full, -the send blocks until their is space to enqueue the message -(this is interruptible via a timeout mechanism). - -A receive right designates a queue and authorizes the holder to -dequeue messages from the queue, and to create send and send-once -rights. - -Send and send-once rights designate a queue and authorize the -hold to enqueue messages (in the case of a send-once right, -a single message). Enqueuing a message is equivalent to -[[invoke|invoking]] a capability. - -Send and receive rights are named using local names. Each -task has associated with it a port [[address_space]]. A ports -are addressed via this table. Each task thus has its own -private [[naming_context]] for ports. - -Ports can be [[delegate]]d in an [[IPC]] message. When the -receiver dequeues the message, the right is made available -to it. - -A [[thread]] can only block receiving on a single port. To work -around this, the concept of a port set was introduced. A receive -right can be added to (at most) one port set. When a thread -receives from a port set, it dequeues from any of the ports that -has a message available. +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[Mach]] *port*s are [[capabilities|capability]], and are also essentially +similar to [[UNIX]] pipes. They are unforgeable communication channels, +implemented by kernel queues. + +Each port has associated with it one *receive right* and one or more *send +right*s and *send-once right*s. That is, there is one receiver and one or more +senders -- a unidirectional communication channel. Only with the corresponding +port right, access to a port is possible; this is enforced by Mach. + +The kernel queue can hold a number of [[message]]s. Once the queue is full, +the send blocks until there is space to enqueue the message (this is +interruptible via a timeout mechanism). + +A receive right [[designates|designation]] a queue and authorizes the holder to +dequeue messages from the queue, and to create send and send-once rights. + +Send and send-once rights designate a queue and authorize the hold to enqueue +messages (in the case of a send-once right, a single message). Enqueuing a +message is equivalent to [[invoke|invoking]] a capability. + +Ports are automatically destroyed when there is no associated port right to +them. + +Mach knows what port rights belong to each task, but [[thread]]s that running +in the context of a task refer to ports by means of send and receive rights +that are named using local *port names*. These port names are plain integers, +like [[UNIX file descriptors|unix/file_descriptor]]. Only these local names +can be used by [[thread]]s for invoking operations on ports, threads do not +deal with port rights directly. + +For that, each task has associated with it a *port address space*, or *port +name space*. All ports are addressed via this table. Each task thus has its +own private [[naming_context]] for port rights. + +So, the picture is that after obtaining a port send right, the client uses a +port name to send [[message]]s to the port, or exactly one message if it's a +send-once right. These messages are (probably) queued and when the server task +tries to receive messages by having a [[thread]] use its port receive right, it +gets the message(s). This is called [[IPC]]. + +Port rights themselves can be [[delegate]]d in a [[message]], too. When the +receiver dequeues the message, the right is made available to it. + +The delivery of [[message]]s is reliable and strictly ordered. When a +[[thread]] sends messages *1* and *2*, it is guaranteed that the receiving +[[task]] will catch them in the same order. Of course, there can be +intermediate messages that are sent by other threads. + +Ports are objects that are implemented by the [[kernel]], and they are +kernel-protected resources: they are unforgeable, and there is no way for a +[[task]] to do anything with a port unless it have corresponding port right. + +Due to this, ports are globally unique. This makes them ideal for constituting +system-wide *object references*. (Fruther reading: +{{$capability#wikipedia_object-capability_model}}.) For example, the [[RPC]] +system as used by the GNU Hurd works by invoking *methods* on such object +references. The available methods are defined in [[hurd/interface]] files, and +are processes by the [[MIG]] tool. + +Invoking an operation on a port does not transfer the current execution control +to the receiver, but instead is an asynchronous operation. For this, and +especially in a [[RPC]] system, the sender may include a *reply port* using a +send-once right, and synchronize (block) on that one. + + +# Port Set + +A [[thread]] can only block receiving on a single port. To work around this, +the concept of a *port set* was introduced. A receive right can be added to +(at most) one port set. These port sets look like port receive rights, but +cannot be passed to other tasks, and there are additional operations for adding +and removing port receive rights. + +When a server process' thread receives from a port set, it dequeues exactly one +message from any of the ports that has a message available in its queue. + +This concept of port sets is also the facility that makes convenient +implementation of [[UNIX's `select` system call|glibc/select]] possible. diff --git a/microkernel/mach/rpc.mdwn b/microkernel/mach/rpc.mdwn index 72acfaa0..3615fc12 100644 --- a/microkernel/mach/rpc.mdwn +++ b/microkernel/mach/rpc.mdwn @@ -1,15 +1,28 @@ -[[!meta copyright="Copyright © 2007, 2008 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2002, 2003, 2007, 2008, 2010, 2011, 2012 Free +Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -[[General_information|/rpc]] about RPC. +Read about the [[general concept of a *remote procedure call* (RPC)|/rpc]]. Uses Mach's [[IPC]] [[mechanism]]. -Stub code generated by [[MIG]]. +The [[port]] abstraction allows RPCs to be executed on another computer +transparently. This can be implemented with user [[task]]s, but there is an +implementation in the kernel possible, too, which is called *NORMA*, but is not +avilable in [[GNU Mach|gnumach]]. + +The RPC stub code is generated by [[MIG]] to send appropriate [[message]]s. + + +# Tracing + + * [[hurd/debugging/rpctrace]] + + * [[open_issues/librpci]] diff --git a/microkernel/mach/rpc/discussion.mdwn b/microkernel/mach/rpc/discussion.mdwn new file mode 100644 index 00000000..00e4a012 --- /dev/null +++ b/microkernel/mach/rpc/discussion.mdwn @@ -0,0 +1,117 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + + +# IRC, freenode, #hurd, 2011-06-11 + + <antrik> I don't think we have a precendence case of Mach initiating RPCs + to userspace tasks + <braunr> well mach regularly sends RPCs to external pagers + <antrik> hm, right + <antrik> anyways, the ds_ in device.defs is for use *inside* Mach, not for + the userspace interface + <braunr> what makes you think so ? + <antrik> several things + <antrik> not least the fact that without zhengda's modifications, the + device handling never calls out to userspace for all I know + <braunr> hm, it does + <braunr> for async I/O + <braunr> when the kernel has finished its I/O, it calls + ds_device_read_reply/ds_device_write_reply + <antrik> I see + <antrik> I never quite understood the _reply stuff + <braunr> although i wonder how mig is supposed to forge those names + <antrik> braunr: it isn't + <antrik> braunr: there is a separate device_reply.defs + <antrik> braunr: and it sets a *userprefix* of ds_ + <antrik> rather than a serverprefix + <braunr> i saw, yes + <braunr> ah right + <antrik> so ds still refers to the in-Mach device server, not anything + userspace + <braunr> so this is where the patch is supposed to introduce the + device_intr_notify RPC + <antrik> or at least that's my understanding... + <braunr> no, it doesn't refer to in-mach servers + <braunr> it really forges the right rpcs to be called by mach + <antrik> the definition of "RPC" is rather unclear here + <braunr> why ? + <braunr> mach has its own mach_msg() call for kernel-to-user messaging + <antrik> yes, but this is used only to send the reply message for the RPC + earlier initiated by userspace AIUI + <antrik> it doesn't look like there is any special RPC for async I/O + <braunr> yes, because this is the only use case they had + <braunr> hence the name "reply" + <braunr> intr_notify isn't a reply, but it uses the same mechanism + <braunr> these are declared as simpleroutine + <antrik> sure. but the fact that it isn't a reply message, but rather + initiates a new RPC, changes things from MiG point of view I believe + <antrik> right, as there is no reply to the reply :-) + <braunr> :) + <braunr> a simpleroutine is how to turn an rpc into a simple ipc + <antrik> I know + <antrik> so in _reply, we pretend that the reply is actually a new RPC, + with server and client roles reversed, and no reply + <antrik> (this is actually rather kludgy... apparently MIG has no real + notion of async replies) + <braunr> i don't understand what you mean + <braunr> simpleroutine is the explicit solution for async replies + <braunr> as stated in + http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig.ps + <braunr> it's not a new rpc with roles reversed + <braunr> it's not a reply either + <antrik> it might be an explicit solution for that, but it still seems + kludgy :-) + <braunr> i don't see why :/ + <braunr> would you have expected something like an option to create both + sync and async versions ? + <antrik> because it requires an extra .defs file + <antrik> yes + <braunr> ok + <braunr> well this seems cumbersome to me :) + <braunr> i prefer the simpleroutine approach + <braunr> but i agree this seems odd since mach has a high level ipc api + <antrik> anyways, my point is that the ds_ in device_reply.defs still + refers to the Mach side of things + <braunr> npnth: which package fails to build ? + <antrik> though a userspace process that actually handles the replies in an + async fashion will of course need some kind of device server too, just + like the DDE stuff... + <antrik> though naming it ds_ is confusing IMHO, because of the name clash + with the device server in Mach + <braunr> hm again, i fail to see why + <braunr> ds_ just means device_server + <braunr> and as most things in mach, it can be in kernel or not + <braunr> i mean, this is an interface prefix, i don't refer to an actual + single instance of a "device server" out there + <antrik> oh, right... DDE implements the Mach device protocol, so it *does* + do the ds_ part... but that makes the interrupt notification stuff even + more confusing + <braunr> hm + <braunr> because it provides a ds_device_intr_notify() which will never be + used, just to completely implement the interface ? + <antrik> yeah, that's what I suspect... + <braunr> sounds likely + <antrik> the device interface actually has two parts: one for "generic" + RPCs on the master device port, and one for device-specific RPCs. DDE + implements the latter, and uses the former... + <antrik> they live in separate places though I think: the individual device + RPCs are implemented in libmachdev, while the intr_ stuff is used in + libddekit probably + <braunr> it would be hairy to build otherwise + <antrik> so we *really* need to know what component npnth gets the error + with + <antrik> braunr: nah, not really. that's why we always have a separate + prefix for the server routines in Hurd RPCs + <braunr> right, i really need to read about mig again + <antrik> it's pretty normal for a translator to both implement and use an + interface diff --git a/microkernel/mach/task.mdwn b/microkernel/mach/task.mdwn new file mode 100644 index 00000000..c03c6a14 --- /dev/null +++ b/microkernel/mach/task.mdwn @@ -0,0 +1,23 @@ +[[!meta copyright="Copyright © 2002, 2003, 2010 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +A Mach *task* is a collection of resources, a [[virtual_address_space]], and a +[[port name space|port]]. They depend on [[thread]]s for executing program +code: a task alone has no means to do so. + +Switching from one task to another one involves doing a *context switch*, which +is usually not a cheap operation, as it involves switching the hardware's idea +of the memory layout ([[virtual_address_space]]), amongst others. + +Mach tasks are distinct from [[UNIX processes|unix/process]] in that they +provide less facilities. In processes, there are [[unix/signal]]s, process / +group / session IDs, [[unix/file_descriptor]]s and many other things. Tasks +are used for resource allocation and sharing; they are *resource container*s. diff --git a/microkernel/mach/thread.mdwn b/microkernel/mach/thread.mdwn new file mode 100644 index 00000000..e27bb117 --- /dev/null +++ b/microkernel/mach/thread.mdwn @@ -0,0 +1,37 @@ +[[!meta copyright="Copyright © 2002, 2003, 2010 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +A Mach *thread* belongs to exactly one [[task]], and is the means of execution. +The task supplies the resources. + +Mach threads are implemented inside the [[kernel]], as opposed to other +systems' user-level thread packages. + +A thread (theoretically) runs concurrently with all the other threads of a +system. If the system provides several processors, they can be used for +simultaneously running either several threads of the same task, or several +threads of different tasks. <!-- This is called SMP; the processors use +*shared memory*. --> [[!tag open_issue_documentation]] <!-- This needs a new +page, also covering Mach's `processor_set`s, and non-SMP, but still +multiprocessor systems. --> (But this is currently not support in [[GNU +Mach|gnumach]].) + +It is easy for the kernel to switch execution from one thread to another one +inside the same task: essentially, it only involves exchanging a few processor +registers' state. + +Threads have scheduling parameters and maintain various statistics about +themselves. + +On GNU/Hurd, APIs for Mach threads and thereabouts are provided by the +[[hurd/libthreads]] (cthreads), and [[libpthread]] (POSIX Threads) packages. + +A task backing a thread is the basis for a [[UNIX process|unix/process]]. diff --git a/microkernel/mach/virtual_address_space.mdwn b/microkernel/mach/virtual_address_space.mdwn new file mode 100644 index 00000000..97bc5f6b --- /dev/null +++ b/microkernel/mach/virtual_address_space.mdwn @@ -0,0 +1,36 @@ +[[!meta copyright="Copyright © 2002, 2003, 2010 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +*Virtual address space*s in Mach define the valid virtual addresses that can be +used by [[thread]]s under execution in the [[task]] that owns that address +space. Each task has only one address space and each address space belongs to +only one task. So when we want to name an address space (for example, in the +Mach API) we name it by the task it belongs to. + +These address spaces are divided into *pages*. Each page has individual +properties like *access rights* (*read* / *write* / *execute*), *inheritance +attributes* (*no inheritance* / *copy* / *share*) and some other system +properties. Page manipulation is optimized to help moving large blocks of data +from one address space to another, for example when one thread provides data to +another thread -- *client / server* technology. + +Memory ranges of pages that can be controlled as a whole are called +*[[memory_object]]*s. + +*Wired pages* are those that cannot be [[paged out|external_pager_mechanism]]. +For example, Mach itself is a task with its own address space and threads, and +all of its pages are wired. + +*Precious pages* are those that must not be discarded silently when they are +clean and memory is needed. For example, a memory manager that shares memory +across a network could not restore a page if it is silently discarded because +it is unmodified. This is not valid for the well-known [[pager +managers|external_pager_mechanism]] that use disks as backing store. |