No subject
Fri Mar 12 14:08:07 EST 2010
are better because if they fail they rebuild quicker. Make sure and turn o=
n
the integrity checks on the RAID in the controller. I check on a weekly
basis early in the morning because the performs gets hammered during the
integrity scan. Discovering a problem in the RAID sooner than later enable=
s
you to deal with hardware loss before you lose data.
Another good idea is to allow for one or two hot-spares so the RAID can
automatically rebuild as soon as a hard drive fails (not if but when).
Another feature that I have found that makes a big difference on performanc=
e
is to turn on the write-cache in the RAID controllers.
My experience is that the 3WARE controllers have worked excellently for me
(they have since been purchased by LSI). I have had negative experiences
with built-in Intel controllers that are built into the motherboards of man=
y
computers these days. The 3WARE controller is also setup to send me emails
when it has issues.
I also continue to have excellent experience with Ubuntu 10.04 Software
RAID. It's very fast and reliable and more flexible than the hardware
controller counter-parts. Also much cheaper. :)
Blessings in the LORD!
--=20
Jeffrey Thompson
President & CEO of JANOAH, INC.
Alpharetta, GA
404-563-7074 x 100
jeffrey at janoah.net
Web: http://janoah.net
Twitter: http://twitter.com/janoahinc
Facebook: http://facebook.com/janoah
"where the Spirit of the Lord is, there is liberty" -- 2 Corinthians 3:17
2010/5/21 Bobby Stewart <bStewart at brentwoodbaptist.com>
> Chris Green at Solerant gave me the following considerations:
>
>
>
> =93If you want to do DFS or similar Windows technologies your file server
> will need to be Windows and not a NAS.=94
>
>
>
> Yes, we are using DFS. This is important.
>
>
>
> =93NFS is not block level. While it is the highest performance network fi=
le
> system it is not VMFS and therefore does not benefit from much of the
> technology in vSphere.=94
>
>
>
> Though most virtualizers here are VMware aficionados, I am not yet
> persuaded. My initial investigation into the pricing of the Microsoft
> management technologies shows a considerable savings over the VMware
> offerings even with their newly available academic pricing for charities.=
I
> still have some detailed feature comparisons ahead. I will start a new
> thread inviting those of you with specific experience with the current
> versions of both offerings to send up your comments.
>
>
>
> Great stuff! Thanks Chris. Keep those cards and letters coming!
>
>
>
> Bobby
>
>
>
> *From:* discuss-bounces at itdiscuss.org [mailto:
> discuss-bounces at itdiscuss.org] *On Behalf Of *Bobby Stewart
> *Sent:* Friday, May 21, 2010 12:31 PM
>
> *To:* IT Discussion Forum
> *Subject:* Re: [itdiscuss] Off to the Races - To LUN or not to LUN...
>
>
>
> Also see the paper: Comparison of Storage Protocol Performance in VMware
> vSphere=99 4<http://www.vmware.com/files/pdf/perf_vsphere_storage_protoco=
ls.pdf>
>
>
>
> =93All four storage protocols for shared storage on ESX [FC, Hardware iSC=
SI,
> Software iSCSI and NFS] are shown to be capable of achieving throughput
> levels that are only limited by the capabilities of the storage array and
> the connection between it and the ESX server. ESX shows excellent
> scalability by maintaining these performance levels in cases of heavy
> consolidation. For CPU cost, Fibre Channel and Hardware iSCSI are more
> efficient than Software iSCSI and NFS. However, when CPU resources are n=
ot
> a bottleneck, Software iSCSI and NFS can also be part of a high-performan=
ce
> solution.=94
>
>
>
> My question is to the conceivable benefits of using NFS/CIFS for some
> storage device functions and iSCSI for others from the same device vs. th=
e
> performance overall. To me, it=92s beginning to look like it wouldn=92t b=
e a
> significant issue for performance. The aspect of not needing to license a=
nd
> setup a virtual machine for file service and potentially other functions =
is
> in the plus column of the cost/benefit analysis.
>
>
>
> Bobby
>
>
>
> *From:* discuss-bounces at itdiscuss.org [mailto:
> discuss-bounces at itdiscuss.org] *On Behalf Of *Bobby Stewart
> *Sent:* Friday, May 21, 2010 11:48 AM
> *To:* IT Discussion Forum
> *Subject:* Re: [itdiscuss] Off to the Races - To LUN or not to LUN...
>
>
>
> Here=92s some real data:
>
>
>
> From VMware vSphere=99 4: Exchange Server=AE on NFS, iSCSI, and Fibre Cha=
nnel<http://www.vmware.com/files/pdf/vsphere_perf_exchange-storage-protocol=
s.pdf>
>
>
>
> =93All three tested protocols [FC, iSCSI and NFS] have shown to be able =
to
> support 16,000 Exchange users in both the Heavy and Double Heavy workload=
s.
> These workloads stressed a 16-core HP DL 580 server to approximately 60
> percent CPU utilization and pushed an average of over 5,000 IOPS to the
> attached NetApp storage array. This throughput was maintained while the
> average latency stayed under 220 milliseconds and 95th percentile latenc=
y
> remained under one second. Fibre Channel provided the best performance wi=
th
> the lowest CPU utilization. NFS and iSCSI performance was comparable with
> slightly longer latencies, and slightly higher CPU utilization.=94
>
>
>
> According to the white paper from VMware, the performance differential is
> marginal. This paper was based on NetApp. Is there any reason to believe
> that other vendors offering similar products would not have similar resul=
ts?
>
>
>
>
>
> Bobby
>
>
>
> *From:* discuss-bounces at itdiscuss.org [mailto:
> discuss-bounces at itdiscuss.org] *On Behalf Of *Bobby Stewart
> *Sent:* Friday, May 21, 2010 11:04 AM
> *To:* IT Discussion Forum
> *Subject:* [itdiscuss] Off to the Races - To LUN or not to LUN...
>
>
>
> As we proceed in our virtualization quest we are trying to cover all the
> basses. For me, this has been a thought provoking process as I entertain =
the
> various technologies available. This is my second post on this topic to s=
eek
> the advice of the community as we go.
>
>
>
> Most, if not all, current SAN solutions for the SMB market provide iSCSI
> capabilities for accessing the SAN at a block level. This will require th=
at
> a device accessing the SAN will access a LUN set up on the SAN for the da=
ta
> therein (stored VMs, actual data stores, etc.).
>
>
>
> Many newer SAN offerings provide alternate methods of accessing data on t=
he
> SAN other than iSCSI. These typically appear as file based access
> principally via CIFS and NFS thus making the SAN a combo SAN/NAS.
> Virtualization purists will decry this tactic suggesting that to do so wi=
ll
> impact performance. My own thoughts, unfounded thought they may be, are
> along these lines:
>
> a) virtualization infrastructures access files located on storage
> media that are the basis of the virtualized servers (VMs)
>
> b) in a traditional iSCSI based system a host OS must be created to
> provide that file infrastructure
>
> c) a VM host accessing these files on a system that already provide=
s
> the file infrastructure will have a resulting reduction in resource usage
>
>
>
> It would seem, then, that a virtual system utilizing a high performance
> file based architecture hosted by the storage device is something to be
> desired in a virtualization environment. A possible down side would be th=
at
> the storage device would be required to provide the resources for that
> architecture and could suffer a resulting performance hit on the iSCSI si=
de.
> Much would depend on the robustness of the device=92s computing platform =
and
> its networking capabilities which would determine its ability to provide
> both block and file architectures simultaneously without performance
> suffering for either or both.
>
>
>
> A secondary benefit will be that portions of the storage infrastructure
> could also be provisioned for direct, file based access for end users thu=
s
> reducing the need to set up virtualized file servers. Obviously, safeguar=
ds
> will be required so that sufficient bandwidth is retained for server
> virtualization and that proper access security is provided.
>
>
>
> So, to paraphrase Shakespeare, =93To LUN or not to LUN =96 that is the
> question.=94 What do you say? What are the risks of using a storage devic=
e
> that would provide both file and block based storage architectures for us=
e
> in a virtualized environment?
>
>
>
> Please try to provide thoughtful, fact based responses where possible.
> Opinions are just that, opinions, and I have many of them that are not wo=
rth
> much.
>
>
>
> Bobby Stewart
> Network Analyst
> Brentwood Baptist Church
> Brentwood, TN
> www.brentwoodbaptist.com
> +1 (615) 324-6149 office
> +1 (615) 830-0012 cell
>
>
>
> _______________________________________________
> it discuss mailing list: discuss at itdiscuss.org
> Mailing List: http://itdiscuss.org/discuss
> Internet Relay Chat: irc://irc.freenode.net/citrt
>
>
--00c09f9725001da2bb04871ebb46
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
Also I would encourage you to keep posting your findings Bobby.=A0 It is in=
formative and helpful.=A0 <br><br>Here are my thoughts on RAID.=A0 This sou=
nds like a level below what you may be thinking, but some of the principles=
may apply to your current project.<br>
<br>From what I have seen, many smaller SAS attached storage (250GB to 500G=
B) are better because if they fail they rebuild quicker.=A0 Make sure and t=
urn on the integrity checks on the RAID in the controller.=A0 I check on a =
weekly basis early in the morning because the performs gets hammered during=
the integrity scan.=A0 Discovering a problem in the RAID sooner than later=
enables you to deal with hardware loss before you lose data.=A0 <br>
Another good idea is to allow for one or two hot-spares so the RAID can aut=
omatically rebuild as soon as a hard drive fails (not if but when).=A0 Anot=
her feature that I have found that makes a big difference on performance is=
to turn on the write-cache in the RAID controllers.<br>
<br>My experience is that the 3WARE controllers have worked excellently for=
me (they have since been purchased by LSI).=A0 I have had negative experie=
nces with built-in Intel controllers that are built into the motherboards o=
f many computers these days.=A0 The 3WARE controller is also setup to send =
me emails when it has issues.<br>
<br>I also continue to have excellent experience with Ubuntu 10.04 Software=
RAID.=A0 It's very fast and reliable and more flexible than the hardwa=
re controller counter-parts.=A0 Also much cheaper.=A0 :)<br><br>Blessings i=
n the LORD!<br>
<br>-- <br>Jeffrey Thompson<br>President & CEO of JANOAH, INC.<br>Alpha=
retta,
GA<br>404-563-7074 x 100<br><a href=3D"mailto:jeffrey at janoah.net">jeffrey@=
janoah.net</a><br>Web:=20
<a href=3D"http://janoah.net">http://janoah.net</a><br>Twitter: <a href=3D"=
http://twitter.com/janoahinc">http://twitter.com/janoahinc</a><br>Facebook:=
=20
<a href=3D"http://facebook.com/janoah">http://facebook.com/janoah</a><br><b=
r>"where the Spirit of the Lord is,=20
there is liberty" -- 2 Corinthians 3:17<br><br><div class=3D"gmail_quo=
te">2010/5/21 Bobby Stewart <span dir=3D"ltr"><<a href=3D"mailto:bStewar=
t at brentwoodbaptist.com">bStewart at brentwoodbaptist.com</a>></span><br><bl=
ockquote class=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt 0.8ex; border-l=
eft: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div link=3D"blue" vlink=3D"purple" lang=3D"EN-US"><div><p class=3D"MsoNorm=
al"><span style=3D"color: rgb(31, 73, 125);">Chris Green at Solerant gave m=
e the following considerations:</span></p><p class=3D"MsoNormal"><span styl=
e=3D"color: rgb(31, 73, 125);">=A0</span></p>
<p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=93If you w=
ant to do DFS or similar Windows technologies your file server will need to=
be Windows and not a NAS.=94</span></p><p class=3D"MsoNormal"><span style=
=3D"color: rgb(31, 73, 125);">=A0</span></p>
<p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">Yes, we are=
using DFS. This is important.</span></p><p class=3D"MsoNormal"><span style=
=3D"color: rgb(31, 73, 125);">=A0</span></p><p class=3D"MsoNormal"><span st=
yle=3D"color: rgb(31, 73, 125);">=93NFS is not block level. While it is the=
highest performance network file system it is not VMFS and therefore does =
not benefit from much of the technology in vSphere.=94</span></p>
<p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=A0</span><=
/p><p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">Though m=
ost virtualizers here are VMware aficionados, I am not yet persuaded. My in=
itial investigation into the pricing of the Microsoft management technologi=
es shows a considerable savings over the VMware offerings even with their n=
ewly available academic pricing for charities. I still have some detailed f=
eature comparisons ahead. I will start a new thread inviting those of you w=
ith specific experience with the current versions of both offerings to send=
up your comments.</span></p>
<p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=A0</span><=
/p><p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">Great st=
uff! Thanks Chris. Keep those cards and letters coming!</span></p><p class=
=3D"MsoNormal">
<span style=3D"color: rgb(31, 73, 125);">=A0</span></p><p class=3D"MsoNorma=
l"><span style=3D"color: rgb(31, 73, 125);">Bobby</span></p><p class=3D"Mso=
Normal"><span style=3D"color: rgb(31, 73, 125);">=A0</span></p><div><div st=
yle=3D"border-width: 1pt medium medium; border-style: solid none none; bord=
er-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color; paddi=
ng: 3pt 0in 0in;">
<p class=3D"MsoNormal"><b><span style=3D"font-size: 10pt;">From:</span></b>=
<span style=3D"font-size: 10pt;"> <a href=3D"mailto:discuss-bounces at itdiscu=
ss.org" target=3D"_blank">discuss-bounces at itdiscuss.org</a> [mailto:<a href=
=3D"mailto:discuss-bounces at itdiscuss.org" target=3D"_blank">discuss-bounces=
@itdiscuss.org</a>] <b>On Behalf Of </b>Bobby Stewart<br>
<b>Sent:</b> Friday, May 21, 2010 12:31 PM<div><div></div><div class=3D"h5"=
><br><b>To:</b> IT Discussion Forum<br><b>Subject:</b> Re: [itdiscuss] Off =
to the Races - To LUN or not to LUN...</div></div></span></p></div></div>
<div><div></div><div class=3D"h5"><p class=3D"MsoNormal">=A0</p><p class=3D=
"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">Also see the paper: <a=
href=3D"http://www.vmware.com/files/pdf/perf_vsphere_storage_protocols.pdf=
" target=3D"_blank">Comparison of Storage Protocol Performance in VMware vS=
phere=99 4</a> </span></p>
<p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=A0</span><=
/p><p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=93All f=
our storage protocols for shared storage on ESX [FC, Hardware iSCSI, Softwa=
re iSCSI and NFS] are shown to be capable of achieving throughput levels th=
at are only limited by the capabilities of the storage array and the connec=
tion between it and the ESX server. ESX shows excellent scalability by main=
taining these performance levels in cases of heavy consolidation. For CPU c=
ost, Fibre Channel and Hardware iSCSI are more efficient than Software iSCS=
I=A0 and NFS. However, when CPU resources are not a bottleneck, Software iS=
CSI and NFS can also be part of a high-performance solution.=94</span></p>
<p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=A0</span><=
/p><p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">My quest=
ion is to the conceivable benefits of using NFS/CIFS for some storage devic=
e functions and iSCSI for others from the same device vs. the performance o=
verall. To me, it=92s beginning to look like it wouldn=92t be a significant=
issue for performance. The aspect of not needing to license and setup a vi=
rtual machine for file service and potentially other functions is in the pl=
us column of the cost/benefit analysis.</span></p>
<p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=A0</span><=
/p><div><p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">Bob=
by</span></p></div><p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73,=
125);">=A0</span></p>
<div><div style=3D"border-width: 1pt medium medium; border-style: solid non=
e none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-=
color; padding: 3pt 0in 0in;"><p class=3D"MsoNormal"><b><span style=3D"font=
-size: 10pt;">From:</span></b><span style=3D"font-size: 10pt;"> <a href=3D"=
mailto:discuss-bounces at itdiscuss.org" target=3D"_blank">discuss-bounces at itd=
iscuss.org</a> [mailto:<a href=3D"mailto:discuss-bounces at itdiscuss.org" tar=
get=3D"_blank">discuss-bounces at itdiscuss.org</a>] <b>On Behalf Of </b>Bobby=
Stewart<br>
<b>Sent:</b> Friday, May 21, 2010 11:48 AM<br><b>To:</b> IT Discussion Foru=
m<br><b>Subject:</b> Re: [itdiscuss] Off to the Races - To LUN or not to LU=
N...</span></p></div></div><p class=3D"MsoNormal">=A0</p><p class=3D"MsoNor=
mal">
<span style=3D"color: rgb(31, 73, 125);">Here=92s some real data: </span></=
p><p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=A0</span=
></p><p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">From <=
a href=3D"http://www.vmware.com/files/pdf/vsphere_perf_exchange-storage-pro=
tocols.pdf" target=3D"_blank">VMware vSphere=99 4: Exchange Server=AE on NF=
S, iSCSI, and Fibre Channel</a> </span></p>
<p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=A0</span><=
/p><p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=93All t=
hree tested protocols [FC, iSCSI and NFS] =A0have shown to be able to suppo=
rt 16,000 Exchange users in both the Heavy and Double Heavy workloads. Thes=
e workloads stressed a 16-core HP DL 580 server to approximately 60 percent=
CPU utilization and pushed an average of over 5,000 IOPS to the attached N=
etApp storage array. This throughput was maintained while the average laten=
cy stayed under 220=A0 milliseconds and 95th percentile latency remained un=
der one second. Fibre Channel provided the best performance with the lowest=
CPU utilization. NFS and iSCSI performance was comparable with slightly lo=
nger latencies, and slightly higher CPU utilization.=94</span></p>
<p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=A0</span><=
/p><p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">Accordin=
g to the white paper from VMware, the performance differential is marginal.=
This paper was based on NetApp. Is there any reason to believe that other =
vendors offering similar products would not have similar results?</span></p=
>
<p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=A0</span><=
/p><p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=A0</spa=
n></p><div><p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=
Bobby</span></p>
</div><p class=3D"MsoNormal"><span style=3D"color: rgb(31, 73, 125);">=A0</=
span></p><div><div style=3D"border-width: 1pt medium medium; border-style: =
solid none none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-=
use-text-color; padding: 3pt 0in 0in;">
<p class=3D"MsoNormal"><b><span style=3D"font-size: 10pt;">From:</span></b>=
<span style=3D"font-size: 10pt;"> <a href=3D"mailto:discuss-bounces at itdiscu=
ss.org" target=3D"_blank">discuss-bounces at itdiscuss.org</a> [mailto:<a href=
=3D"mailto:discuss-bounces at itdiscuss.org" target=3D"_blank">discuss-bounces=
@itdiscuss.org</a>] <b>On Behalf Of </b>Bobby Stewart<br>
<b>Sent:</b> Friday, May 21, 2010 11:04 AM<br><b>To:</b> IT Discussion Foru=
m<br><b>Subject:</b> [itdiscuss] Off to the Races - To LUN or not to LUN...=
</span></p></div></div><p class=3D"MsoNormal">=A0</p><p class=3D"MsoNormal"=
>As we proceed in our virtualization quest we are trying to cover all the b=
asses. For me, this has been a thought provoking process as I entertain the=
various technologies available. This is my second post on this topic to se=
ek the advice of the community as we go.</p>
<p class=3D"MsoNormal">=A0</p><p class=3D"MsoNormal">Most, if not all, curr=
ent SAN solutions for the SMB market provide iSCSI capabilities for accessi=
ng the SAN at a block level. This will require that a device accessing the =
SAN will access a LUN set up on the SAN for the data therein (stored VMs, a=
ctual data stores, etc.).</p>
<p class=3D"MsoNormal">=A0</p><p class=3D"MsoNormal">Many newer SAN offerin=
gs provide alternate methods of accessing data on the SAN other than iSCSI.=
These typically appear as file based access principally via CIFS and NFS t=
hus making the SAN a combo SAN/NAS. Virtualization purists will decry this =
tactic suggesting that to do so will impact performance. My own thoughts, u=
nfounded thought they may be, are along these lines: </p>
<p><span>a)<span style=3D"font: 7pt "Times New Roman";">=A0=A0=A0=
=A0=A0 </span></span>virtualization infrastructures access files located on=
storage media that are the basis of the virtualized servers (VMs)</p><p><s=
pan>b)<span style=3D"font: 7pt "Times New Roman";">=A0=A0=A0=A0=
=A0 </span></span>in a traditional iSCSI based system a host OS must be cre=
ated to provide that file infrastructure</p>
<p><span>c)<span style=3D"font: 7pt "Times New Roman";">=A0=A0=A0=
=A0=A0=A0 </span></span>a VM host accessing these files on a system that al=
ready provides the file infrastructure will have a resulting reduction in r=
esource usage</p>
<p class=3D"MsoNormal">=A0</p><p class=3D"MsoNormal">It would seem, then, t=
hat a virtual system utilizing a high performance file based architecture h=
osted by the storage device is something to be desired in a virtualization =
environment. A possible down side would be that the storage device would be=
required to provide the resources for that architecture and could suffer a=
resulting performance hit on the iSCSI side. Much would depend on the robu=
stness of the device=92s computing platform and its networking capabilities=
which would determine its ability to provide both block and file architect=
ures simultaneously without performance suffering for either or both.</p>
<p class=3D"MsoNormal">=A0</p><p class=3D"MsoNormal">A secondary benefit wi=
ll be that portions of the storage infrastructure could also be provisioned=
for direct, file based access for end users thus reducing the need to set =
up virtualized file servers. Obviously, safeguards will be required so that=
sufficient bandwidth is retained for server virtualization and that proper=
access security is provided.</p>
<p class=3D"MsoNormal">=A0</p><p class=3D"MsoNormal">So, to paraphrase Shak=
espeare, =93To LUN or not to LUN =96 that is the question.=94 What do you s=
ay? What are the risks of using a storage device that would provide both fi=
le and block based storage architectures for use in a virtualized environme=
nt?</p>
<p class=3D"MsoNormal">=A0</p><p class=3D"MsoNormal">Please try to provide =
thoughtful, fact based responses where possible. Opinions are just that, op=
inions, and I have many of them that are not worth much.</p><p class=3D"Mso=
Normal">
=A0</p><p class=3D"MsoNormal">Bobby Stewart<br>Network Analyst<br>Brentwood=
Baptist Church<br>Brentwood, TN<br><a href=3D"http://www.brentwoodbaptist.=
com" target=3D"_blank">www.brentwoodbaptist.com</a><br>+1 (615) 324-6149 of=
fice<br>
+1 (615) 830-0012 cell</p><p class=3D"MsoNormal">=A0</p></div></div></div><=
/div><br>_______________________________________________<br>
it discuss mailing list: <a href=3D"mailto:discuss at itdiscuss.org">discuss at i=
tdiscuss.org</a><br>
Mailing List: <a href=3D"http://itdiscuss.org/discuss" target=3D"_blank">ht=
tp://itdiscuss.org/discuss</a><br>
Internet Relay Chat: irc://<a href=3D"http://irc.freenode.net/citrt" target=
=3D"_blank">irc.freenode.net/citrt</a><br>
<br></blockquote></div><br><br clear=3D"all"><br>
--00c09f9725001da2bb04871ebb46--
More information about the discuss
mailing list