Search:   Namespaces Discussions .NET v1.1 Feedback

file I/O Messages   Related Types This message was discovered on microsoft.public.dotnet.general. Responses highlighted in red are from those people who are likely to be able to contribute good, authoratitive information to this discussion. They include Microsoft employees, MVP's and others who IMHO contribute well to these kinds of discussions. Post a new message to this list... shiva (VIP) Inorder to handle large files over 100 MB, is there a way in dotnet to open only certain bytes of the file by buffer ? It is shaky if the entire file is opened and splitted. Instead is there a way or any alternatives on opening small chunk of the file. Reply to this message...        Peter Rilling I don't think there is any different between opening a 1M file or 100M file if you use a stream. Streams are designed with buffers and will only read in as much information as you want. So, you could open your file using FileStream and seek to a specified location, then read as many bytes as you want. Now, if you need more complex processing like getting the 200th row of some file that are delimited by newlines, then that is something that you might have to process yourself or rely on something like a Stream reader. "shiva" <Click here to reveal e-mail address> wrote in message news:Click here to reveal e-mail address... [Original message clipped] Reply to this message...        shiva (VIP) Thank you for the feedback Sir. Well, i am just trying to print a 30MB file and its shaky. It gets hogged sometimes amd prints sometimes as if it works when we have luck. Unless or otherwise something is wrong with the following code. There is lot more transactions to do than just printing, but it is skaky right here when printing. [code] Private Sub Button1_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles Button1.Click 'Open a file for reading Dim FILENAME As String Dim FileContents As String Dim iCount As Integer iCount = 1 FILENAME = "c:\\Filereader\\File1.txt" Try 'Get a StreamReader class that can be used to read the file Dim objStreamReader As StreamReader objStreamReader = File.OpenText(FILENAME) 'Read Line By Line While True FileContents = CType(objStreamReader.ReadLine, String) If Not FileContents = Nothing Then Response.Write(iCount & "Begin Line1" & "<br>") Response.Write(FileContents) Response.Write("<br>" & iCount & "End Line1" & "<br>") iCount = iCount + 1 'lblRawOutput.Text &= FileContents & vbCrLf Else : Exit While End If End While 'Set the text of the file to a Web control 'lblRawOutput.Text = contents objStreamReader.Close() Catch ex As Exception Response.Write(ex.Message) End Try End Sub [/code] "Peter Rilling" wrote: [Original message clipped] Reply to this message...        shiva (VIP) Thank you for the feedback Sir. Well, i am just trying to print a 30MB file and its shaky. It gets hogged sometimes amd prints sometimes as if it works when we have luck. Unless or otherwise something is wrong with the following code. There is lot more transactions to do than just printing, but it is skaky right here when printing. Private Sub Button1_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles Button1.Click 'Open a file for reading Dim FILENAME As String Dim FileContents As String Dim iCount As Integer iCount = 1 FILENAME = "c:\\Filereader\\File1.txt" Try 'Get a StreamReader class that can be used to read the file Dim objStreamReader As StreamReader objStreamReader = File.OpenText(FILENAME) 'Read Line By Line While True FileContents = CType(objStreamReader.ReadLine, String) If Not FileContents = Nothing Then Response.Write(iCount & "Begin Line1" & "<br>") Response.Write(FileContents) Response.Write("<br>" & iCount & "End Line1" & "<br>") iCount = iCount + 1 'lblRawOutput.Text &= FileContents & vbCrLf Else : Exit While End If End While 'Set the text of the file to a Web control 'lblRawOutput.Text = contents objStreamReader.Close() Catch ex As Exception Response.Write(ex.Message) End Try End Sub "Peter Rilling" wrote: [Original message clipped] Reply to this message...        Andrew Faust "shiva" <Click here to reveal e-mail address> wrote in message news:Click here to reveal e-mail address... [Original message clipped] ReadLine returns a string already, so there is no reason to re-caste it. [Original message clipped] It doesn't look like your speed issues are due to the way you are working with the files, but rather with what you are doing with all the data. What is this Response object you are working with? I'm assuming it's something like a web response. If this is the case then it can take quite a while to send the 30 MB of data via the network. Even on a local network it can take a while depending on network configuration and current traffic loads. Also you have 'lblRawOutput.Text &= FileContents & vbCrLf commented out. However, if your tests were done with this uncommented it could have a drastic performance hit for large files, as the larger a string gets, the longer it takes to concatenate more data on to the end of it. Andrew Faust Reply to this message...        RyanKrypton (VIP) I am definately not an expert in these things but when I was making my program that encrypts any type of file, I found that working with myltiple threads makes a huge difference. You said that the program seems a little shaky when you are handling large files, so try to use one thread to manaage the printinge etc... and the other (the first one initialised )to continue running to maintain the programm in a state in which it can continue responding to the Operating System. Even if this may not be the right answer, it may stiil help toe programm to maintain stability. Ryan "shiva" wrote: [Original message clipped] Reply to this message...        shiva (VIP) Thank you for the feedback Sir. Well, i am just trying to print a 30MB file and its shaky. It gets hogged sometimes amd prints sometimes as if it works when we have luck. Unless or otherwise something is wrong with the following code. There is lot more transactions to do than just printing, but it is skaky right here when printing. [code] Private Sub Button1_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles Button1.Click 'Open a file for reading Dim FILENAME As String Dim FileContents As String Dim iCount As Integer iCount = 1 FILENAME = "c:\\Filereader\\File1.txt" Try 'Get a StreamReader class that can be used to read the file Dim objStreamReader As StreamReader objStreamReader = File.OpenText(FILENAME) 'Read Line By Line While True FileContents = CType(objStreamReader.ReadLine, String) If Not FileContents = Nothing Then Response.Write(iCount & "Begin Line1" & "<br>") Response.Write(FileContents) Response.Write("<br>" & iCount & "End Line1" & "<br>") iCount = iCount + 1 'lblRawOutput.Text &= FileContents & vbCrLf Else : Exit While End If End While 'Set the text of the file to a Web control 'lblRawOutput.Text = contents objStreamReader.Close() Catch ex As Exception Response.Write(ex.Message) End Try End Sub [/code] "Peter Rilling" wrote: [Original message clipped] Reply to this message...        Cor Ligthert Shiva, Are you sure you want it byte by byte, used are often Unicodecharters which used two bytes. Than the encoding class will probably one of the first things too look at. http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemtextencodingclasstopic.asp I hope this helps? Copr "shiva" <Click here to reveal e-mail address> > Inorder to handle large files over 100 MB, is there a way in dotnet to open [Original message clipped] Reply to this message...        shiva (VIP) Dear Cor, For example if the file is a 30MB file, i am looking for opening for 1MB at a time. Even if i open the 30MB and split it into 1MB, since the memory is accumulated it runs out of memory easily. I am looking if there is a way i can open the 1MB at one time, instead of opening the 30MB by means of buffering or any other alternatives. Thanks much Shiva "Cor Ligthert" wrote: [Original message clipped] Reply to this message...        Andrew Faust "shiva" <Click here to reveal e-mail address> wrote in message news:Click here to reveal e-mail address... [Original message clipped] You can't open only 1 MB at a time. You either have a file open or you don't. Size is irrelevant. However, you can read in to memory only a small portion at a time. Andrew Faust Reply to this message...        Cor Ligthert Shiva, Tell us first what type of file it is, text, a kind of own database, are there char in it etc. or maybe it are pictures. Now it is just shooting in the dark. Cor Reply to this message...        shiva (VIP) Hi Cor, Its a regular text file, but the formats are pretty complex like a EDI file. There are no pictures or any kind of picture data in the file. Its a regular file Header1<Information>endofline Detail1<Information>endofline Detail2<Information>endofline Header2<Information>endofline Detail1<Information>endofline Detail2<Information>endofline Detail3<Information>endofline Header3<Information>endofline Detail1<Information>endofline Header4<Information>endofline Detail1<Information>endofline Detail2<Information>endofline Detail3<Information>endofline I understand if i read only a 1MB out of the 30MB i may not get an entire record, but i can come up with some other algorithm for that. But the main question is, is there a way i can open 1MB out of the 30MB. Thank you Shiva "Cor Ligthert" wrote: [Original message clipped] Reply to this message...        Cor Ligthert Shiva, What is than the reason you do than not proces it line by line? http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpguide/html/cpconReadingTextFromFile.asp Cor "shiva" <Click here to reveal e-mail address> schreef in bericht news:Click here to reveal e-mail address... [Original message clipped] Reply to this message...        shiva (VIP) Dear Cor, The reason i don't want to do line by line was it runs out of memory. Dear Andrew, I removed the Ctype and the Response.write but the difference in performance is very less. Thank you Siva "Cor Ligthert" wrote: [Original message clipped] Reply to this message...        Cor Ligthert Shiva, When you process it line by line why it can than not be in a kind of pseudo Private rowcounter = 0 Process 10000rows(not endoffile) Procedure Process10000rows do while not endoffile For rowcounter to 10000 read line into whatever next When this can fit and you cannot make this in real code tell than? Cor "shiva" <Click here to reveal e-mail address> [Original message clipped] Reply to this message...     System.EventArgs System.IO.File System.IO.FileStream System.IO.StreamReader Ad MBR BootFX Best-of-breed application framework for .NET projects, developed by Matthew Baxter-Reynolds and MBR IT

Copyright © Matthew Baxter-Reynolds 2001-2008. '.NET 247 Software Development Services' is a trading style of MBR IT Solutions Ltd. Contact Us - Terms of Use - Privacy Policy - www.dotnet247.com