如何解决如何使用 Java 使用 csv 文件导出大量数据近 100 万条数据
我正在尝试从 Postgresql 数据库中导出数据，其中我必须导出的数据量接近 100 万。我尝试了各种方法，但没有得到解决方案。即使我使用邮递员来调用 API，我已经编写了导出 csv 的 API，邮递员正在关闭。我正在使用 react.js下载但它在那里加载了几个小时。我正在发布导出代码
public String populateCsvReport(SearchDto searchDto){
List<DetailRecord> myDetailRecord = itsCustomrepo.getDetail(searchDto);
StringWriter sw = new StringWriter();
try(CSVPrinter csvPrinter = new CSVPrinter(sw,CSVFormat.DEFAULT.withHeader("supplier Number"
"supplier name"........
)){
myDetailRecord.forEach(mydetail->{
csvPrinter.printRecord(
mydetail.getsuplNum(),mydetail.getsuplName(),......................
)
});
return myDetailRecord;
现在我也尝试将代码更改为
myDetailRecord.forEach(mydetail->{
mydetail.getsuplNum(),......................
});
csvPrinter.printRecord(
myDetailRecord
);
但这并没有对我的代码产生影响。
在我的控制器中，我正在这样做
@Getmapping(path="/get-export-detail/csv"){
public RespnseEntity<String> generateMydetailExport(SearchDto searchDto){
return ResponseEntity.ok()
.header("Content-disposition","attachment;fileName="+"myDetails.csv")
.contentType(MediaType.parseMediaType("text/csv"))
.body(callingService.populateCSVForDetail(searchDto));
const exportOnClick=()=>{
callingDetailsService.export(param)
.then(response)=>{
let mime = "test/csv";
let fileName = "myDetail.csv";
util.downloadFile(response.data,fileName,mime);
@Repository
public class ItsCustomrepo{
@PersistanceContext
private EntityManager entityManager;
public List<DetailRecord> getDetail(List<SearchCriteria> params){
List<DetailRecord> listofDetail = new ArrayList<>();
CriteriaBuilder cb = entityManager.getCriteriaBuilder();
CriteriaQuery<DetailEntity> cQuery = cb.createquery(DetailEntity.class);
Root<DetailEntity> rootE = cQuery.from(DetailEntity.class);
String sqlQuery = "select ............."
if(params.size()>0){
for(SearchCriteria param:params){
if(param.getValue()!=null && param.getValue()!=""){
if(param.getoperation().equalIgnoreCase(CriteriaOperationEnum.GREATER_THAN_OR_EQUAL.getoperation()){
if(rootE.get(param.getKey()).getJavaType()==LocalDate.class){
}else if(param.getoperation().equalIgnoreCase(CriteriaOperationEnum.LESS_THAN_OR_EQUAL.getoperation()
//some op
}else{ if(param.getoPeration().equalsIgnoreCase(CriteriaOperationEnum.LIKE.getoperation())){
//some op
}
}
Query query = entityManager.createNativeQuery(sqlQuery);
List<Object[]> objectList = query.getResultList();
int count = objectList.size();
objectList.forEach(glObject->{
DetailRecord detailRecord = DetailRecord.builder()
.supl_num(glObject[0])
...................
listofDetail .add(detailRecord);
});
return listofDetail;
我的代码很简单，因为我不明白它在哪里失败，我在运行查询时检查数据库的计数并且速度很快，而且我可以在调试代码时看到顺利进入控制器，但之后它挂了好几个小时。我尝试过使用 opencsv、apache-poi 等。无法理解它在哪里失败，请有人帮助我。
解决方法
这里是一些示例代码，它使用两种方法生成一些 CSV。第一个与您的类似——它获取列表中的所有行，然后创建 csv。第二种方法更“流式”，因为它会在从数据库中获取行时立即写出行。有 100 万行，这有很大的不同。
import org.h2.jdbcx.JdbcDataSource;
import org.springframework.jdbc.core.JdbcTemplate;
import java.util.List;
import java.util.function.Consumer;
public class CsvSample {
static class Player {
int id;
String name;
int teamId;
Player(int id,String name,int temId) {
this.id = id;
this.name = name;
this.teamId = temId;
}
}
interface PlayerRepo {
void save(Player player);
List<Player> findPlayers(int teamId);
int processPlayers(int teamId,Consumer<Player> callback);
}
static class SimplePlayerRepo implements PlayerRepo {
JdbcTemplate jdbc;
SimplePlayerRepo(JdbcTemplate jdbc) {
this.jdbc = jdbc;
this.jdbc.execute("create table if not exists Player(id int primary key,name varchar(30),team int)");
}
@Override
public void save(Player player) {
int n = jdbc.update(
"update Player set name=?,team=? where id=?",player.name,player.teamId,player.id);
if (n == 0) {
jdbc.update(
"insert into Player(name,team,id) values (?,?,?)",player.id);
}
}
@Override
public List<Player> findPlayers(int teamId) {
return jdbc.query(
"select id,name,team from Player where team=?",(rs,n) -> new Player(rs.getInt(1),rs.getString(2),rs.getInt(3)),teamId);
}
@Override
public int processPlayers(int teamId,Consumer<Player> callback) {
return jdbc.query(
"select id,rs -> {
int n = 0;
while (rs.next()) {
Player p = new Player(rs.getInt(1),rs.getInt(3));
callback.accept(p);
}
return n;
},teamId);
}
}
public static void main(String[] args) throws Exception {
JdbcDataSource dataSource = new JdbcDataSource();
dataSource.setUrl("jdbc:h2:mem:csvsample;DB_CLOSE_DELAY=-1");
PlayerRepo repo = new SimplePlayerRepo(new JdbcTemplate(dataSource));
// add some players
repo.save(new Player(1,"Kobe",1));
repo.save(new Player(2,"LeBron",1));
repo.save(new Player(3,"Shaq",1));
repo.save(new Player(4,"Kareem",1));
repo.save(new Player(5,"Magic",1));
repo.save(new Player(6,"Larry",2));
repo.save(new Player(7,"Jason",2));
// generate CSV from List
repo.findPlayers(1).forEach(player -> {
System.out.println(player.id + "," + player.name);
});
System.out.println("----");
// generate CSV with callback
repo.processPlayers(1,player -> {
System.out.println(player.id + "," + player.name);
});
}
}
因此，在您的情况下，我会向您的存储库类添加一个方法。它应该包含您的 getDetail 方法中的所有逻辑，直到您到达显示 Query query = entityManager.createNativeQuery(sqlQuery); 的行：
public int processSearchResults(List<SearchCriteria> params,Consumer<DetailRecord> callback){
// instead of this:
// Query query = entityManager.createNativeQuery(sqlQuery);
Session session = entityManager.unwrap(Session.class);
return session.doWork(new Work() {
@Override
public void execute(Connection connection) throws SQLException {
Statement stmt = connection.createQuery();
ResultSet rs = stmt.executeQuery(sqlQuery);
int n = 0;
while (rs.next()) {
DetailRecord detailRecord = DetailRecord.builder()
.supl_num(rs.getObject(1))
// .....
.build();
callback.accept(detailRecord);
n++;
}
rs.close();
stmt.close();
return n;
}
});
}
然后，您的 generateMydetailExport 可能如下所示：
@Getmapping(path="/get-export-detail/csv",produces="text/csv") {
public void generateMydetailExport(SearchDto searchDto,PrintWriter out) {
CSVPrinter csvPrinter = new CSVPrinter(out,CSVFormat.DEFAULT.withHeader("Supplier Number",/* ... */));
itsCustomRepo.processSearchResults(searchDto,detail -> {
csvPrinter.printRecord(
mydetail.getSuplNum(),mydetail.getSuplName(),// .....
);
});
}
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。
