iqbalrifai
/
Aegisub
mirror of https://github.com/odrling/Aegisub
1
0
Fork 0
Code Issues Releases Wiki Activity
Aegisub/src/charset_detect.cpp

63 lines
2.4 KiB
C++
Raw Normal View History

Switch to using libaegisub for character set detection. There are some bugs here but it seems to be more consistent than the last version. There are two remaining issues left: when the character set is unknown we need to provide a full list to choose from. The second: if the file is detected as US-ASCII we need to give a parse error to the user if we run into problems. Right now we'll load the file fine and disable a lot of options.. with no message to the user. Originally committed to SVN as r4370.
2010-05-29 04:25:19 +02:00
// Copyright (c) 2010, Amar Takhar
Instead of falling back to your local charset, Aegisub will now use the "universalchardet" library (the one used by the Mozilla project) to autodetect the character set of non-Unicode files. Originally committed to SVN as r1028.
2007-04-08 08:01:41 +02:00
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are met:
//
// * Redistributions of source code must retain the above copyright notice,
// this list of conditions and the following disclaimer.
// * Redistributions in binary form must reproduce the above copyright notice,
// this list of conditions and the following disclaimer in the documentation
// and/or other materials provided with the distribution.
// * Neither the name of the Aegisub Group nor the names of its contributors
// may be used to endorse or promote products derived from this software
// without specific prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
// POSSIBILITY OF SUCH DAMAGE.
//
Switch all headers to using Doxygen and cleanup contact info * Swap old email + website address with 'Aegisub Project http://www.aegisub.org/' * Set categories for all files (jfs) * Add descriptions for each file (jfs) * Add $Id$ keyword Originally committed to SVN as r3310.
2009-07-29 07:43:02 +02:00
// Aegisub Project http://www.aegisub.org/
/// @file charset_detect.cpp
/// @brief Wrapper around text encoding detection library
/// @ingroup utility
///
Instead of falling back to your local charset, Aegisub will now use the "universalchardet" library (the one used by the Mozilla project) to autodetect the character set of non-Unicode files. Originally committed to SVN as r1028.
2007-04-08 08:01:41 +02:00
Mostly purge wxWidgets from non-UI stuff Use boost::filesystem::path for all paths, and std::string for all other strings, converting to/from wxString as close to the actual uses of wx as possible. Where possible, replace the uses of non-UI wxWidgets functionality with the additions to the standard library in C++11, or the equivalents in boost. Move the path token management logic to libaegisub (and rewrite it in the process). Add a basic thread pool based on asio and std::thread to libaegisub. This touches nearly every file in the project and a nontrivial amount of code had to be rewritten entirely, so there's probably a lot of broken stuff.
2013-01-04 16:01:50 +01:00
#include "charset_detect.h"
Fix all the headers in *.cpp, this includes: * Wrapping all headers that are in agi_pre.h with AGI_PRE. * Sorting alphabetically. Originally committed to SVN as r3515.
2009-09-10 15:06:40 +02:00
Mostly purge wxWidgets from non-UI stuff Use boost::filesystem::path for all paths, and std::string for all other strings, converting to/from wxString as close to the actual uses of wx as possible. Where possible, replace the uses of non-UI wxWidgets functionality with the additions to the standard library in C++11, or the equivalents in boost. Move the path token management logic to libaegisub (and rewrite it in the process). Add a basic thread pool based on asio and std::thread to libaegisub. This touches nearly every file in the project and a nontrivial amount of code had to be rewritten entirely, so there's probably a lot of broken stuff.
2013-01-04 16:01:50 +01:00
#include "compat.h"
Add a whole slew of missing includes, prior to r3509 config_unix.h used to include stdwx.h, now that I'm fixing the headers I removed it which caused issues when not using precompiled headers. Originally committed to SVN as r3512.
2009-09-10 07:25:25 +02:00
Switch to using libaegisub for character set detection. There are some bugs here but it seems to be more consistent than the last version. There are two remaining issues left: when the character set is unknown we need to provide a full list to choose from. The second: if the file is detected as US-ASCII we need to give a parse error to the user if we run into problems. Right now we'll load the file fine and disable a lot of options.. with no message to the user. Originally committed to SVN as r4370.
2010-05-29 04:25:19 +02:00
#include <libaegisub/charset.h>
Display the full list of supported character sets if detection fails catastrophically
2013-02-05 04:17:04 +01:00
#include <libaegisub/charset_conv.h>
Mostly purge wxWidgets from non-UI stuff Use boost::filesystem::path for all paths, and std::string for all other strings, converting to/from wxString as close to the actual uses of wx as possible. Where possible, replace the uses of non-UI wxWidgets functionality with the additions to the standard library in C++11, or the equivalents in boost. Move the path token management logic to libaegisub (and rewrite it in the process). Add a basic thread pool based on asio and std::thread to libaegisub. This touches nearly every file in the project and a nontrivial amount of code had to be rewritten entirely, so there's probably a lot of broken stuff.
2013-01-04 16:01:50 +01:00
#include <wx/arrstr.h>
#include <wx/choicdlg.h>
#include <wx/intl.h>
Add a whole slew of missing includes, prior to r3509 config_unix.h used to include stdwx.h, now that I'm fixing the headers I removed it which caused issues when not using precompiled headers. Originally committed to SVN as r3512.
2009-09-10 07:25:25 +02:00
Simplify charset detection Originally committed to SVN as r4419.
2010-06-03 22:31:43 +02:00
namespace CharSetDetect {
Instead of falling back to your local charset, Aegisub will now use the "universalchardet" library (the one used by the Mozilla project) to autodetect the character set of non-Unicode files. Originally committed to SVN as r1028.
2007-04-08 08:01:41 +02:00
Mostly purge wxWidgets from non-UI stuff Use boost::filesystem::path for all paths, and std::string for all other strings, converting to/from wxString as close to the actual uses of wx as possible. Where possible, replace the uses of non-UI wxWidgets functionality with the additions to the standard library in C++11, or the equivalents in boost. Move the path token management logic to libaegisub (and rewrite it in the process). Add a basic thread pool based on asio and std::thread to libaegisub. This touches nearly every file in the project and a nontrivial amount of code had to be rewritten entirely, so there's probably a lot of broken stuff.
2013-01-04 16:01:50 +01:00
std::string GetEncoding(agi::fs::path const& filename) {
Switch to a maintained fork of universalchardet
2016-02-08 04:52:45 +01:00
auto encoding = agi::charset::Detect(filename);
if (!encoding.empty())
return encoding;
Display the full list of supported character sets if detection fails catastrophically
2013-02-05 04:17:04 +01:00
Switch to a maintained fork of universalchardet
2016-02-08 04:52:45 +01:00
auto choices = agi::charset::GetEncodingsList<wxArrayString>();
Mostly purge wxWidgets from non-UI stuff Use boost::filesystem::path for all paths, and std::string for all other strings, converting to/from wxString as close to the actual uses of wx as possible. Where possible, replace the uses of non-UI wxWidgets functionality with the additions to the standard library in C++11, or the equivalents in boost. Move the path token management logic to libaegisub (and rewrite it in the process). Add a basic thread pool based on asio and std::thread to libaegisub. This touches nearly every file in the project and a nontrivial amount of code had to be rewritten entirely, so there's probably a lot of broken stuff.
2013-01-04 16:01:50 +01:00
int choice = wxGetSingleChoiceIndex(
_("Aegisub could not narrow down the character set to a single one.\nPlease pick one below:"),
_("Choose character set"),
choices);
if (choice == -1) throw agi::UserCancelException("Cancelled encoding selection");
Switch to a maintained fork of universalchardet
2016-02-08 04:52:45 +01:00
return from_wx(choices[choice]);
Improved window for detection of charset, but still doesn't solve cp1258 issue. Originally committed to SVN as r1789.
2008-01-20 08:24:04 +01:00
}
Simplify charset detection Originally committed to SVN as r4419.
2010-06-03 22:31:43 +02:00
}